May 15 10:18:32.521703 np0000174319 systemd[1]: Started devstack@neutron-ovn-maintenance-worker.service - Devstack devstack@neutron-ovn-maintenance-worker.service. May 15 10:18:32.907271 np0000174319 neutron-ovn-maintenance-worker[104616]: /opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/lockutils.py:41: EventletDeprecationWarning: May 15 10:18:32.907271 np0000174319 neutron-ovn-maintenance-worker[104616]: Eventlet is deprecated. It is currently being maintained in bugfix mode, and May 15 10:18:32.907271 np0000174319 neutron-ovn-maintenance-worker[104616]: we strongly recommend against using it for new projects. May 15 10:18:32.907271 np0000174319 neutron-ovn-maintenance-worker[104616]: If you are already using Eventlet, we recommend migrating to a different May 15 10:18:32.907271 np0000174319 neutron-ovn-maintenance-worker[104616]: framework. For more detail see May 15 10:18:32.907271 np0000174319 neutron-ovn-maintenance-worker[104616]: https://eventlet.readthedocs.io/en/latest/asyncio/migration.html May 15 10:18:32.907271 np0000174319 neutron-ovn-maintenance-worker[104616]: import eventlet May 15 10:18:34.110697 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.common.config [-] Logging enabled! May 15 10:18:34.111163 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-ovn-maintenance-worker version 28.1.0.dev221 May 15 10:18:34.111163 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-ovn-maintenance-worker --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini {{(pid=104616) setup_logging /opt/stack/neutron/neutron/common/config.py:124}} May 15 10:18:34.111341 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.server.ovn_maintenance [-] OVN maintenance process starting... May 15 10:18:34.111468 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG oslo_concurrency.lockutils [-] Acquiring lock "plugin-directory" by "neutron_lib.plugins.directory._create_plugin_directory" {{(pid=104616) inner /opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/lockutils.py:506}} May 15 10:18:34.111620 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG oslo_concurrency.lockutils [-] Lock "plugin-directory" acquired by "neutron_lib.plugins.directory._create_plugin_directory" :: waited 0.000s {{(pid=104616) inner /opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/lockutils.py:519}} May 15 10:18:34.111729 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG oslo_concurrency.lockutils [-] Lock "plugin-directory" "released" by "neutron_lib.plugins.directory._create_plugin_directory" :: held 0.000s {{(pid=104616) inner /opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/lockutils.py:538}} May 15 10:18:34.111808 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG oslo_concurrency.lockutils [-] Acquiring lock "manager" by "neutron.manager.NeutronManager._create_instance" {{(pid=104616) inner /opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/lockutils.py:506}} May 15 10:18:34.111913 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG oslo_concurrency.lockutils [-] Lock "manager" acquired by "neutron.manager.NeutronManager._create_instance" :: waited 0.000s {{(pid=104616) inner /opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/lockutils.py:519}} May 15 10:18:34.112010 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.manager [-] Loading core plugin: ml2 May 15 10:18:34.444273 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_create 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:34.444523 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_update 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:34.444523 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_delete 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:34.445304 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_create 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:34.445430 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_update 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:34.445491 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_delete 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:34.486725 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_create 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:34.486725 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_update 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:34.486725 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_delete 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:34.608388 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_create 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:34.608595 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_update 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:34.608595 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_delete 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:34.644082 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_create 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:34.644212 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_update 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:34.644212 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_delete 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:34.645919 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_create 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:34.646073 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_update 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:34.646073 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_delete 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:34.722596 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: agent after_create 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:34.722804 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: agent after_update 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:34.722804 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: segment precommit_create 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:34.722912 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: network precommit_delete 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:34.723431 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: segment before_delete 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:34.781901 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > network before_delete 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:34.781901 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: agent before_delete 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:34.781901 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > network before_create 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:34.782434 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port before_create 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:34.782434 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port before_update 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:34.782434 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > segment precommit_create 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:34.782434 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > segment precommit_delete 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:34.782434 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > segment after_create 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:34.782896 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > segment after_delete 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:34.782896 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > network after_delete 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:34.782896 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > network precommit_delete 0, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:34.782896 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port provisioning_complete 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:34.782896 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_create 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:34.783535 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy after_delete 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:34.783535 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > agent after_update 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:34.783535 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > subnet after_delete 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:34.783535 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > subnet precommit_delete 0, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:34.783535 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_update 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:34.783983 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_delete 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:34.783983 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > segment_host_mapping after_create 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:34.783983 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > agent after_delete 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:34.783983 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_create 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:34.783983 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_update 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:34.784356 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_delete 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:34.784356 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.quota.resource_registry [-] Tracking information for resource: network configured {{(pid=104616) set_tracked_resource /opt/stack/neutron/neutron/quota/resource_registry.py:203}} May 15 10:18:34.784356 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.quota.resource_registry [-] Tracking information for resource: port configured {{(pid=104616) set_tracked_resource /opt/stack/neutron/neutron/quota/resource_registry.py:203}} May 15 10:18:34.784356 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.quota.resource_registry [-] Tracking information for resource: subnet configured {{(pid=104616) set_tracked_resource /opt/stack/neutron/neutron/quota/resource_registry.py:203}} May 15 10:18:34.784356 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.quota.resource_registry [-] Tracking information for resource: subnetpool configured {{(pid=104616) set_tracked_resource /opt/stack/neutron/neutron/quota/resource_registry.py:203}} May 15 10:18:34.784356 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.quota.resource_registry [-] Tracking information for resource: security_group configured {{(pid=104616) set_tracked_resource /opt/stack/neutron/neutron/quota/resource_registry.py:203}} May 15 10:18:34.784356 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.quota.resource_registry [-] Tracking information for resource: security_group_rule configured {{(pid=104616) set_tracked_resource /opt/stack/neutron/neutron/quota/resource_registry.py:203}} May 15 10:18:34.784356 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.plugins.ml2.managers [-] Configured type driver names: ['local', 'flat', 'vlan', 'geneve'] May 15 10:18:34.785001 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.plugins.ml2.drivers.type_flat [-] Allowable flat physical_network names: ['public'] May 15 10:18:34.785621 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.plugins.ml2.drivers.type_local [-] ML2 LocalTypeDriver initialization complete May 15 10:18:34.786190 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.plugins.ml2.drivers.type_vlan [-] Network VLAN ranges: OrderedDict({'public': [(1, 4094)]}) May 15 10:18:34.786260 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.plugins.ml2.managers [-] Loaded type driver names: ['flat', 'geneve', 'local', 'vlan'] May 15 10:18:34.786400 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.plugins.ml2.managers [-] Registered types: dict_keys(['flat', 'geneve', 'local', 'vlan']) May 15 10:18:34.786479 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.plugins.ml2.managers [-] Project network_types: ['geneve'] May 15 10:18:34.786596 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.plugins.ml2.managers [-] Configured extension driver names: ['port_security', 'qos'] May 15 10:18:34.787124 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.plugins.ml2.managers [-] Loaded extension driver names: ['port_security', 'qos'] May 15 10:18:34.787197 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.plugins.ml2.managers [-] Registered extension drivers: ['port_security', 'qos'] May 15 10:18:34.787252 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.plugins.ml2.managers [-] Configured mechanism driver names: ['ovn'] May 15 10:18:34.824826 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.plugins.ml2.managers [-] Loaded mechanism driver names: ['ovn'] May 15 10:18:34.824826 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.plugins.ml2.managers [-] Registered mechanism drivers: ['ovn'] May 15 10:18:34.825057 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.plugins.ml2.managers [-] No mechanism drivers provide segment reachability information for agent scheduling. May 15 10:18:34.825563 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > floatingip before_response 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:34.825716 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port before_response 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:34.850140 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.plugins.ml2.managers [-] Initializing driver for type 'flat' May 15 10:18:34.850140 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.plugins.ml2.drivers.type_flat [-] ML2 FlatTypeDriver initialization complete May 15 10:18:34.850358 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.plugins.ml2.managers [-] Initializing driver for type 'geneve' May 15 10:18:34.850358 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.plugins.ml2.drivers.type_tunnel [-] geneve ID ranges: [(1, 65536)] May 15 10:18:34.898518 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO dbcounter [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Registered counter for database neutron May 15 10:18:34.915216 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG oslo_db.sqlalchemy.engines [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=104616) _check_effective_sql_mode /opt/stack/data/venv/lib/python3.12/site-packages/oslo_db/sqlalchemy/engines.py:325}} May 15 10:18:35.349084 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG dbcounter [-] [104616] Writer thread running {{(pid=104616) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:102}} May 15 10:18:35.567397 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.plugins.ml2.managers [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Initializing driver for type 'local' May 15 10:18:35.567581 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.plugins.ml2.managers [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Initializing driver for type 'vlan' May 15 10:18:35.698525 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.plugins.ml2.drivers.type_vlan [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] VlanTypeDriver initialization complete May 15 10:18:35.698713 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.plugins.ml2.managers [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Initializing extension driver 'port_security' May 15 10:18:35.698713 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.plugins.ml2.extensions.port_security [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] PortSecurityExtensionDriver initialization complete May 15 10:18:35.698773 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.plugins.ml2.managers [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Initializing extension driver 'qos' May 15 10:18:35.698875 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.plugins.ml2.extensions.qos [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] QosExtensionDriver initialization complete {{(pid=104616) initialize /opt/stack/neutron/neutron/plugins/ml2/extensions/qos.py:31}} May 15 10:18:35.698952 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.plugins.ml2.managers [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Initializing mechanism driver 'ovn' May 15 10:18:35.699040 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Starting OVNMechanismDriver May 15 10:18:35.700628 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Subscribe: > process before_spawn 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:35.700760 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Subscribe: > process after_init 55550000, True {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:35.700851 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Subscribe: > segment after_create 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:35.700949 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Subscribe: > segment after_create 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:35.701037 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Subscribe: > segment after_delete 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:35.701119 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Subscribe: > allowed_address_pair before_create 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:35.701205 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Subscribe: > security_group precommit_create 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:35.701300 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Subscribe: > security_group after_update 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:35.701387 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Subscribe: > security_group after_create 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:35.701471 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Subscribe: > security_group precommit_delete 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:35.701552 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Subscribe: > security_group after_delete 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:35.701638 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Subscribe: > security_group_rule precommit_create 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:35.701737 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Subscribe: > security_group_rule after_create 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:35.701821 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Subscribe: > security_group_rule before_delete 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:35.701891 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Subscribe: > address_group after_create 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:35.701970 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Subscribe: > address_group after_update 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:35.702200 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Subscribe: > address_group after_delete 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:35.702271 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Subscribe: > qos_plugin after_init 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:35.702459 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Subscribe: > trunk_plugin after_init 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:35.702644 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Subscribe: > logging-plugin after_init 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:35.703184 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.services.logapi.drivers.ovn.driver [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] OVN logging driver registered May 15 10:18:35.704548 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.plugins.ml2.plugin [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] neutron.plugins.ml2.plugin.Ml2Plugin method _start_rpc_notifiers called with arguments () {} {{(pid=104616) wrapper /opt/stack/data/venv/lib/python3.12/site-packages/oslo_log/helpers.py:77}} May 15 10:18:35.704766 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Subscribe: > port after_create 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:35.704903 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Subscribe: > port after_update 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:35.704903 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Subscribe: > port after_delete 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:35.705040 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Subscribe: > subnet after_create 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:35.705116 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Subscribe: > subnet after_update 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:35.705178 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Subscribe: > subnet after_delete 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:35.705238 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Subscribe: > network after_create 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:35.705309 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Subscribe: > network after_update 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:35.705370 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Subscribe: > network after_delete 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:35.705447 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Subscribe: > security_group after_create 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:35.705551 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Subscribe: > security_group after_update 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:35.705627 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Subscribe: > security_group after_delete 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:35.705755 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Subscribe: > security_group_rule after_create 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:35.705839 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Subscribe: > security_group_rule after_update 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:35.705902 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Subscribe: > security_group_rule after_delete 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:35.705998 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Subscribe: > address_group after_create 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:35.706086 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Subscribe: > address_group after_update 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:35.706158 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Subscribe: > address_group after_delete 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:35.706218 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.plugins.ml2.ovo_rpc [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] ML2 OVO RPC backend initialized. {{(pid=104616) __init__ /opt/stack/neutron/neutron/plugins/ml2/ovo_rpc.py:117}} May 15 10:18:35.707135 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.quota [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loaded quota_driver: . May 15 10:18:35.707244 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.plugins.ml2.plugin [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Modular L2 Plugin initialization complete May 15 10:18:35.707385 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.manager [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading services supported by the core plugin {{(pid=104616) _load_services_from_core_plugin /opt/stack/neutron/neutron/manager.py:170}} May 15 10:18:35.707488 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.plugins.ml2.managers [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Got port-security extension from driver 'port_security' May 15 10:18:35.707792 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.manager [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading service plugins: ['ovn-router', 'auto_allocate', 'tag', 'timestamp', 'network_ip_availability', 'flavors', 'revisions'] {{(pid=104616) _load_service_plugins /opt/stack/neutron/neutron/manager.py:195}} May 15 10:18:35.707890 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.manager [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading Plugin: ovn-router May 15 10:18:35.716113 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Subscribe: > router before_delete 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:35.716219 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Subscribe: port before_update 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:35.716292 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Subscribe: port after_update 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:35.716706 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Subscribe: > port precommit_delete 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:35.716706 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Subscribe: port before_update 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:35.716706 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Subscribe: port before_delete 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:35.716706 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Subscribe: > router precommit_create 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:35.716923 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Subscribe: > router precommit_create 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:35.716923 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Subscribe: > router precommit_update 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:35.716923 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Subscribe: subnet after_update 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:35.717060 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.quota.resource_registry [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Tracking information for resource: router configured {{(pid=104616) set_tracked_resource /opt/stack/neutron/neutron/quota/resource_registry.py:203}} May 15 10:18:35.717182 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.quota.resource_registry [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Tracking information for resource: floatingip configured {{(pid=104616) set_tracked_resource /opt/stack/neutron/neutron/quota/resource_registry.py:203}} May 15 10:18:35.717360 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.services.ovn_l3.plugin [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Starting OVNL3RouterPlugin May 15 10:18:35.717418 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Subscribe: > port_forwarding_plugin after_init 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:35.717588 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Subscribe: > router before_create 55550100, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:35.717663 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Subscribe: > router precommit_delete 55550100, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:35.717765 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Subscribe: > router precommit_create 55550100, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:35.717858 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Subscribe: > router precommit_update 55550100, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:35.720398 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.services.provider_configuration [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Service providers = [] {{(pid=104616) parse_service_provider_opt /opt/stack/neutron/neutron/services/provider_configuration.py:170}} May 15 10:18:35.720521 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.db.servicetype_db [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Adding provider configuration for service L3_ROUTER_NAT {{(pid=104616) add_provider_configuration /opt/stack/neutron/neutron/db/servicetype_db.py:45}} May 15 10:18:35.721723 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Subscribe: > router precommit_create 55549900, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:35.721808 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Subscribe: > router_interface after_create 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:35.721908 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Subscribe: > floatingip after_create 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:35.721968 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Subscribe: > floatingip precommit_create 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:35.722065 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Subscribe: > floatingip precommit_update 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:35.722135 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Subscribe: > floatingip after_delete 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:35.722219 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Subscribe: > floatingip after_status_update 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:35.722301 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Subscribe: > floatingip after_update 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:35.722382 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Subscribe: > port_forwarding after_create 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:35.722467 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Subscribe: > router_interface after_delete 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:35.722552 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Subscribe: > router after_create 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:35.722619 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Subscribe: > router precommit_create 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:35.722730 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Subscribe: > router after_delete 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:35.722810 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Subscribe: > router after_update 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:35.722901 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.services.service_base [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loaded 'neutron.services.ovn_l3.service_providers.ovn.OvnDriver' provider for service L3_ROUTER_NAT {{(pid=104616) load_drivers /opt/stack/neutron/neutron/services/service_base.py:49}} May 15 10:18:35.723010 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Subscribe: > process after_init 55550001, True {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:35.723132 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.manager [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Successfully loaded L3_ROUTER_NAT plugin. Description: L3 Router Service Plugin for basic L3 forwarding using OVN {{(pid=104616) _create_and_add_service_plugin /opt/stack/neutron/neutron/manager.py:239}} May 15 10:18:35.723215 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.manager [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading Plugin: auto_allocate May 15 10:18:35.724074 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Subscribe: network precommit_update 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:35.724148 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron_lib.callbacks.manager [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Subscribe: network precommit_create 55550000, False {{(pid=104616) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:35.724376 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.manager [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Successfully loaded auto-allocated-topology plugin. Description: Auto Allocated Topology - aka get me a network. {{(pid=104616) _create_and_add_service_plugin /opt/stack/neutron/neutron/manager.py:239}} May 15 10:18:35.724451 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.manager [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading Plugin: tag May 15 10:18:35.727351 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.manager [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Successfully loaded TAG plugin. Description: Tag support {{(pid=104616) _create_and_add_service_plugin /opt/stack/neutron/neutron/manager.py:239}} May 15 10:18:35.727443 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.manager [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading Plugin: timestamp May 15 10:18:35.728483 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.manager [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Successfully loaded timestamp plugin. Description: Adds timestamps to Neutron resources with standard attributes {{(pid=104616) _create_and_add_service_plugin /opt/stack/neutron/neutron/manager.py:239}} May 15 10:18:35.728558 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.manager [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading Plugin: network_ip_availability May 15 10:18:35.729795 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.manager [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Successfully loaded network-ip-availability plugin. Description: Provides IP availability data for each network and subnet. {{(pid=104616) _create_and_add_service_plugin /opt/stack/neutron/neutron/manager.py:239}} May 15 10:18:35.729869 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.manager [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading Plugin: flavors May 15 10:18:35.730646 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.manager [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Successfully loaded FLAVORS plugin. Description: Neutron Flavors and Service Profiles manager plugin {{(pid=104616) _create_and_add_service_plugin /opt/stack/neutron/neutron/manager.py:239}} May 15 10:18:35.730725 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.manager [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading Plugin: revisions May 15 10:18:35.731559 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.manager [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Successfully loaded revision_plugin plugin. Description: Adds revision numbers to resources. {{(pid=104616) _create_and_add_service_plugin /opt/stack/neutron/neutron/manager.py:239}} May 15 10:18:35.731690 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG oslo_concurrency.lockutils [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Lock "manager" "released" by "neutron.manager.NeutronManager._create_instance" :: held 1.620s {{(pid=104616) inner /opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/lockutils.py:538}} May 15 10:18:35.731889 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] get_extension_paths = OrderedDict({'/opt/stack/neutron/neutron/extensions': 1}) {{(pid=104616) get_extensions_path /opt/stack/neutron/neutron/api/extensions.py:640}} May 15 10:18:35.731964 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Initializing extension manager. May 15 10:18:35.732191 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: __init__.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.732272 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: __pycache__ {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.732373 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: address_group.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.732710 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Address group" alias="address-group" description="Support address group" updated="2020-07-15T10:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.732989 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loaded extension: address-group May 15 10:18:35.733091 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: address_scope.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.733441 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Address scope" alias="address-scope" description="Address scopes extension." updated="2015-07-26T10:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.733651 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loaded extension: address-scope May 15 10:18:35.733805 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: admin_state_down_before_update.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.734081 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Enforce Router's Admin State Down Before Update Extension" alias="router-admin-state-down-before-update" description="Ensure that the admin state of a router is DOWN (admin_state_up=False) before updating the distributed attribute" updated="2019-07-02T15:56:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.734280 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Extension router-admin-state-down-before-update not supported by any of loaded plugins May 15 10:18:35.734380 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: agent.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.734765 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="agent" alias="agent" description="The agent management extension." updated="2013-02-03T10:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.735198 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loaded extension: agent May 15 10:18:35.735317 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: agent_resources_synced.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.735617 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Agent's Resource View Synced to Placement" alias="agent-resources-synced" description="Stores success/failure of last sync to Placement" updated="2018-12-19T00:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.735888 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Extension agent-resources-synced not supported by any of loaded plugins May 15 10:18:35.736206 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: agent_sort_key.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.736613 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Agents Sort Key" alias="agent_sort_key" description="Enabling the keyword sort_key for sorting in agents." updated="2013-02-03T10:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.736842 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Extension agent_sort_key not supported by any of loaded plugins May 15 10:18:35.736947 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: allowedaddresspairs.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.737571 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Allowed Address Pairs" alias="allowed-address-pairs" description="Provides allowed address pairs" updated="2013-07-23T10:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.737759 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loaded extension: allowed-address-pairs May 15 10:18:35.737863 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: auto_allocated_topology.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.738055 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Auto Allocated Topology Services" alias="auto-allocated-topology" description="Auto Allocated Topology Services." updated="2016-01-01T00:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.738189 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loaded extension: auto-allocated-topology May 15 10:18:35.738268 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: availability_zone.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.738489 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Availability Zone" alias="availability_zone" description="The availability zone extension." updated="2015-01-01T10:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.738612 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loaded extension: availability_zone May 15 10:18:35.738700 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: availability_zone_filter.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.738900 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Availability Zone Filter Extension" alias="availability_zone_filter" description="Add filter parameters to AvailabilityZone resource" updated="2018-06-22T10:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.739025 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Extension availability_zone_filter not supported by any of loaded plugins May 15 10:18:35.739088 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: data_plane_status.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.739322 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Port data plane status extension" alias="data-plane-status" description="Expose status of underlying data plane" updated="2017-01-24T10:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.739483 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Extension data-plane-status not supported by any of loaded plugins May 15 10:18:35.739545 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: default_subnetpools.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.739783 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Default Subnetpools" alias="default-subnetpools" description="Provides ability to mark and use a subnetpool as the default." updated="2016-02-18T18:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.739920 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loaded extension: default-subnetpools May 15 10:18:35.739997 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: dhcpagentscheduler.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.740362 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="DHCP Agent Scheduler" alias="dhcp_agent_scheduler" description="Schedule networks among dhcp agents" updated="2013-02-07T10:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.740475 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loaded extension: dhcp_agent_scheduler May 15 10:18:35.740579 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: dns.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.740763 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="DNS Integration" alias="dns-integration" description="Provides integration with DNS." updated="2015-08-15T18:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.740895 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loaded extension: dns-integration May 15 10:18:35.740968 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: dns_domain_ports.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.741152 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="dns_domain for ports" alias="dns-domain-ports" description="Allows the DNS domain to be specified for a network port." updated="2017-04-24T10:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.741287 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loaded extension: dns-domain-ports May 15 10:18:35.741365 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: dns_integration_domain_keywords.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.741533 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="DNS domain names with keywords allowed" alias="dns-integration-domain-keywords" description="Allows to use keywords like , , and as DNS domain name" updated="2020-06-15T18:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.741667 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loaded extension: dns-integration-domain-keywords May 15 10:18:35.741768 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: dvr.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.741968 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Distributed Virtual Router" alias="dvr" description="Enables configuration of Distributed Virtual Routers." updated="2014-06-1T10:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.742086 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Extension dvr not supported by any of loaded plugins May 15 10:18:35.742153 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: ecmp_routes.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.742343 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Equal Cost Multipath Routing" alias="ecmp_routes" description="Allows traffic to reach thesame destination via multiple different links " updated="2020-11-26T18:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.742479 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Extension ecmp_routes not supported by any of loaded plugins May 15 10:18:35.742559 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: empty_string_filtering.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.742737 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Empty String Filtering Extension" alias="empty-string-filtering" description="Allow filtering by attributes with empty string value" updated="2018-05-01T10:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.742869 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loaded extension: empty-string-filtering May 15 10:18:35.742934 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: expose_l3_conntrack_helper.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.743148 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Expose CT target rules for conntrack helper" alias="expose-l3-conntrack-helper" description="Expose allow adding CT target rules for conntrack helper" updated="2019-04-04T10:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.743252 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Extension expose-l3-conntrack-helper not supported by any of loaded plugins May 15 10:18:35.743342 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: expose_port_forwarding_in_fip.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.743525 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Expose Floating IP port forwardings" alias="expose-port-forwarding-in-fip" description="Expose port forwarding fields in floating IP response" updated="2018-06-28T16:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.743657 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Extension expose-port-forwarding-in-fip not supported by any of loaded plugins May 15 10:18:35.743769 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: external_net.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.743924 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Neutron external network" alias="external-net" description="Adds external network attribute to network resource." updated="2013-01-14T10:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.744054 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loaded extension: external-net May 15 10:18:35.744114 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: extra_dhcp_opt.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.744302 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Neutron Extra DHCP options" alias="extra_dhcp_opt" description="Extra options configuration for DHCP. For example PXE boot options to DHCP clients can be specified (e.g. tftp-server, server-ip-address, bootfile-name)" updated="2013-03-17T12:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.744450 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loaded extension: extra_dhcp_opt May 15 10:18:35.744538 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: extraroute.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.744733 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Neutron Extra Route" alias="extraroute" description="Extra routes configuration for L3 router" updated="2013-02-01T10:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.744857 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loaded extension: extraroute May 15 10:18:35.744928 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: extraroute_atomic.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.745130 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Atomically add/remove extra routes" alias="extraroute-atomic" description="Edit extra routes of a router on server side by atomically adding/removing extra routes" updated="2019-07-10T00:00:00+00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.745258 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loaded extension: extraroute-atomic May 15 10:18:35.745315 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: filter_validation.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.745512 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Filter parameters validation" alias="filter-validation" description="Provides validation on filter parameters." updated="2018-07-04T10:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.745669 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loaded extension: filter-validation May 15 10:18:35.745792 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: fip_pf_description.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.745988 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Floating IP Port Forwarding new attribute description" alias="floating-ip-port-forwarding-description" description="Add a description field to Port Forwarding rules" updated="2019-11-01T10:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.746126 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Extension floating-ip-port-forwarding-description not supported by any of loaded plugins May 15 10:18:35.746197 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: fip_pf_detail.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.746389 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Floating IP port forwarding detail" alias="floating-ip-port-forwarding-detail" description="Allow list floating ip return more port forwarding data, include'id' and 'internal_port_id'" updated="2022-05-06T10:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.746518 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loaded extension: floating-ip-port-forwarding-detail May 15 10:18:35.746588 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: fip_pf_port_range.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.746787 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Floating IP Port Forwarding support port ranges" alias="floating-ip-port-forwarding-port-ranges" description="Add support to port ranges" updated="2020-07-01T10:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.747069 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Extension floating-ip-port-forwarding-port-ranges not supported by any of loaded plugins May 15 10:18:35.747151 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: fip_port_details.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.747338 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Floating IP Port Details Extension" alias="fip-port-details" description="Add port_details attribute to Floating IP resource" updated="2018-04-09T10:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.747475 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loaded extension: fip-port-details May 15 10:18:35.747529 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: flavors.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.747770 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Neutron Service Flavors" alias="flavors" description="Flavor specification for Neutron advanced services." updated="2015-09-17T10:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.747897 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loaded extension: flavors May 15 10:18:35.747967 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: floating_ip_port_forwarding.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.748234 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Floating IP Port Forwarding" alias="floating-ip-port-forwarding" description="Allow port forwarding from floating IP port to an internal port" updated="2018-01-19T10:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.748467 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Extension floating-ip-port-forwarding not supported by any of loaded plugins May 15 10:18:35.748529 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: floatingip_pools.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.748784 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Floating IP Pools Extension" alias="floatingip-pools" description="Provides a floating IP pools API." updated="2018-03-21T10:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.748912 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loaded extension: floatingip-pools May 15 10:18:35.748975 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: ip_allocation.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.749167 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="IP Allocation" alias="ip_allocation" description="IP allocation extension." updated="2016-06-10T23:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.749288 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Extension ip_allocation not supported by any of loaded plugins May 15 10:18:35.749349 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: ip_substring_port_filtering.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.749551 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="IP address substring filtering" alias="ip-substring-filtering" description="Provides IP address substring filtering when listing ports" updated="2017-11-28T09:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.749670 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Extension ip-substring-filtering not supported by any of loaded plugins May 15 10:18:35.749742 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: l2_adjacency.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.749936 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="L2 Adjacency" alias="l2_adjacency" description="Display L2 Adjacency for Neutron Networks." updated="2016-04-12T16:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.750062 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Extension l2_adjacency not supported by any of loaded plugins May 15 10:18:35.750134 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: l3.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.750486 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Neutron L3 Router" alias="router" description="Router abstraction for basic L3 forwarding between L2 Neutron networks and access to external networks via a NAT gateway." updated="2012-07-20T10:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.750627 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loaded extension: router May 15 10:18:35.750720 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: l3_conntrack_helper.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.751220 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="L3 Conntrack helper" alias="l3-conntrack-helper" description="Allow adding CT target rules for conntrack helper" updated="2019-04-04T10:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.751358 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Extension l3-conntrack-helper not supported by any of loaded plugins May 15 10:18:35.751415 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: l3_enable_default_route_bfd.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.751688 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Enable Default Route BFD" alias="enable-default-route-bfd" description="Enables configurable BFD behavior for default routes" updated="2023-02-27T00:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.752110 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loaded extension: enable-default-route-bfd May 15 10:18:35.752110 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: l3_enable_default_route_ecmp.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.752241 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Enable Default Route ECMP" alias="enable-default-route-ecmp" description="Enables configurable ECMP behavior for default routes" updated="2023-02-27T00:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.752437 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loaded extension: enable-default-route-ecmp May 15 10:18:35.752541 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: l3_ext_gw_mode.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.752813 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Neutron L3 Configurable external gateway mode" alias="ext-gw-mode" description="Extension of the router abstraction for specifying whether SNAT should occur on the external gateway" updated="2013-03-28T10:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.752974 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loaded extension: ext-gw-mode May 15 10:18:35.753049 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: l3_ext_ha_mode.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.753244 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="HA Router extension" alias="l3-ha" description="Adds HA capability to routers." updated="2014-04-26T00:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.753377 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loaded extension: l3-ha May 15 10:18:35.753446 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: l3_ext_ndp_proxy.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.753630 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Router support to enable NDP proxy" alias="l3-ext-ndp-proxy" description="Router extension to enable ndp proxy" updated="2020-08-25T00:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.753779 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Extension l3-ext-ndp-proxy not supported by any of loaded plugins May 15 10:18:35.753833 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: l3_extra_gws.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.754023 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Neutron L3 External Gateway Multihoming" alias="external-gateway-multihoming" description="Allow multiple external gateway ports per router" updated="2023-01-18T00:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.754152 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loaded extension: external-gateway-multihoming May 15 10:18:35.754208 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: l3_flavors.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.754394 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Router Flavor Extension" alias="l3-flavors" description="Flavor support for routers." updated="2016-05-17T00:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.754524 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loaded extension: l3-flavors May 15 10:18:35.754595 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: l3_ndp_proxy.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.754862 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="name" alias="l3-ndp-proxy" description="description" updated="2021-06-29T10:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.754984 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Extension l3-ndp-proxy not supported by any of loaded plugins May 15 10:18:35.755051 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: l3_port_ip_change_not_allowed.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.755249 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Prevent L3 router ports IP address change extension" alias="l3-port-ip-change-not-allowed" description="Prevent change of IP address for some L3 router ports" updated="2018-10-09T10:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.755383 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Extension l3-port-ip-change-not-allowed not supported by any of loaded plugins May 15 10:18:35.755438 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: l3agentscheduler.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.755905 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="L3 Agent Scheduler" alias="l3_agent_scheduler" description="Schedule routers among l3 agents" updated="2013-02-07T10:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.756104 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Extension l3_agent_scheduler not supported by any of loaded plugins May 15 10:18:35.756193 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: local_ip.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.756485 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Local IP" alias="local_ip" description="Support Node Local Virtual IP" updated="2021-07-26T10:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.756619 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Extension local_ip not supported by any of loaded plugins May 15 10:18:35.756726 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: logging.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.756951 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Logging API Extension" alias="logging" description="Provides a logging API for resources such as security group and firewall." updated="2017-01-01T10:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.757087 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Extension logging not supported by any of loaded plugins May 15 10:18:35.757142 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: metering.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.757529 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Neutron Metering" alias="metering" description="Neutron Metering extension." updated="2013-06-12T10:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.757690 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Extension metering not supported by any of loaded plugins May 15 10:18:35.757749 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: metering_source_and_destination_fields.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.757973 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Neutron Metering with source and destination filtering" alias="metering_source_and_destination_fields" description="Neutron Metering extension that enables the use of source and destination IP prefixes to create metering label rules." updated="2020-08-21T08:12:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.758078 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Extension metering_source_and_destination_fields not supported by any of loaded plugins May 15 10:18:35.758153 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: multiprovidernet.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.758388 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Multi Provider Network" alias="multi-provider" description="Expose mapping of virtual networks to multiple physical networks" updated="2013-06-27T10:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.758529 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loaded extension: multi-provider May 15 10:18:35.758612 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: netmtu.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.758847 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Network MTU" alias="net-mtu" description="Provides MTU attribute for a network resource." updated="2015-03-25T10:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.758970 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loaded extension: net-mtu May 15 10:18:35.759044 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: netmtu_writable.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.759271 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Network MTU (writable)" alias="net-mtu-writable" description="Provides a writable MTU attribute for a network resource." updated="2017-07-12T00:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.759437 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loaded extension: net-mtu-writable May 15 10:18:35.759476 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: network_availability_zone.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.759669 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Network Availability Zone" alias="network_availability_zone" description="Availability zone support for network." updated="2015-01-01T10:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.759822 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loaded extension: network_availability_zone May 15 10:18:35.759894 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: network_ha.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.760071 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Network HA creation flag" alias="network_ha" description="Network high availability creation flag." updated="2023-04-27T10:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.760204 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Extension network_ha not supported by any of loaded plugins May 15 10:18:35.760273 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: network_ip_availability.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.760511 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Network IP Availability" alias="network-ip-availability" description="Provides IP availability data for each network and subnet." updated="2015-09-24T00:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.760645 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loaded extension: network-ip-availability May 15 10:18:35.760732 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: network_ip_availability_details.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.760972 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Network IP Availability Details Extension" alias="network-ip-availability-details" description="Add ip_availability_details attribute to network-ip-availabilities resource." updated="2025-10-31T00:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.761124 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loaded extension: network-ip-availability-details May 15 10:18:35.761216 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: network_segment_range.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.761477 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Neutron Network Segment Range" alias="network-segment-range" description="Provides support for the network segment range management" updated="2018-11-29T00:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.761614 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Extension network-segment-range not supported by any of loaded plugins May 15 10:18:35.761711 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: pagination.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.761954 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Pagination support" alias="pagination" description="Extension that indicates that pagination is enabled." updated="2016-06-12T00:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.762041 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loaded extension: pagination May 15 10:18:35.762113 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: port_device_profile.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.762306 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Port device profile" alias="port-device-profile" description="Expose the port device profile (Cyborg)" updated="2020-12-17T10:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.762498 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loaded extension: port-device-profile May 15 10:18:35.762601 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: port_hardware_offload_type.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.762855 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Port hardware offload" alias="port-hardware-offload-type" description="Expose port hardware offload extension" updated="2023-05-09T10:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.763042 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Extension port-hardware-offload-type not supported by any of loaded plugins May 15 10:18:35.763134 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: port_hint_ovs_tx_steering.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.763331 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Port hint: Open vSwitch Tx steering" alias="port-hint-ovs-tx-steering" description="Enable Port hint: Open vSwitch Tx steering" updated="2023-01-01T00:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.763523 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Extension port-hint-ovs-tx-steering not supported by any of loaded plugins May 15 10:18:35.763644 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: port_hints.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.763912 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Port hints" alias="port-hints" description="Backend specific port hints to allow backend specific tuning" updated="2023-01-01T00:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.764078 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Extension port-hints not supported by any of loaded plugins May 15 10:18:35.764155 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: port_mac_address_override.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.764386 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Neutron Port MAC address override" alias="port-mac-override" description="Allow overriding the MAC address of a direct-physical Port via the active binding profile" updated="2022-03-18T10:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.764514 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Extension port-mac-override not supported by any of loaded plugins May 15 10:18:35.764582 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: port_mac_address_regenerate.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.764818 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Neutron Port MAC address regenerate" alias="port-mac-address-regenerate" description="Network port MAC address regenerate" updated="2018-05-03T10:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.764975 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loaded extension: port-mac-address-regenerate May 15 10:18:35.765065 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: port_numa_affinity_policy.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.765296 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Port NUMA affinity policy" alias="port-numa-affinity-policy" description="Expose the port NUMA affinity policy" updated="2020-07-08T10:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.765484 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loaded extension: port-numa-affinity-policy May 15 10:18:35.765579 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: port_numa_affinity_policy_socket.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.765785 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Port NUMA affinity policy "socket"" alias="port-numa-affinity-policy-socket" description="Adds "socket" to the supported port NUMA affinity policies" updated="2024-02-09T10:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.765940 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loaded extension: port-numa-affinity-policy-socket May 15 10:18:35.766049 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: port_resource_request.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.766266 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Port Resource Request" alias="port-resource-request" description="Expose resource request to Port" updated="2018-05-08T10:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.766403 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Extension port-resource-request not supported by any of loaded plugins May 15 10:18:35.766476 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: port_resource_request_groups.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.766669 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Port Resource Request Groups" alias="port-resource-request-groups" description="Support requesting multiple groups of resources and traits from the same RP subtree in resource_request" updated="2021-08-02T10:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.766824 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Extension port-resource-request-groups not supported by any of loaded plugins May 15 10:18:35.766898 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: port_trusted_vif.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.767079 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Port trusted vif" alias="port-trusted-vif" description="Expose port 'trusted' attribute in the API" updated="2024-07-10T10:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.767249 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Extension port-trusted-vif not supported by any of loaded plugins May 15 10:18:35.767327 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: portbindings.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.767509 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Port Binding" alias="binding" description="Expose port bindings of a virtual port to external application" updated="2014-02-03T10:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.767647 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loaded extension: binding May 15 10:18:35.767736 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: portbindings_extended.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.767954 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Port Bindings Extended" alias="binding-extended" description="Expose port bindings of a virtual port to external application" updated="2017-07-17T10:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.768089 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loaded extension: binding-extended May 15 10:18:35.768160 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: portsecurity.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.768376 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Port Security" alias="port-security" description="Provides port security" updated="2012-07-23T10:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.768508 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loaded extension: port-security May 15 10:18:35.768579 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: project_id.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.768790 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="project_id field enabled" alias="project-id" description="Extension that indicates that project_id field is enabled." updated="2016-09-09T09:09:09-09:09" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.768848 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loaded extension: project-id May 15 10:18:35.768930 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: providernet.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.769133 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Provider Network" alias="provider" description="Expose mapping of virtual networks to physical networks" updated="2012-09-07T10:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.769286 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loaded extension: provider May 15 10:18:35.769376 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: qinq.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.769582 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="QinQ networks" alias="qinq" description="Expose network 'qinq' attribute in the API. This allows to configure network to allow vlan in vlan configuration using 0x8a88 ethertype (QinQ)." updated="2024-12-02T10:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.769738 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loaded extension: qinq May 15 10:18:35.769813 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: qos.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.770523 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Quality of Service" alias="qos" description="The Quality of Service extension." updated="2015-06-08T10:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.770670 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Extension qos not supported by any of loaded plugins May 15 10:18:35.770759 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: qos_bw_limit_direction.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.770992 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Direction for QoS bandwidth limit rule" alias="qos-bw-limit-direction" description="Allow to configure QoS bandwidth limit rule with specific direction: ingress or egress" updated="2017-04-10T10:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.771159 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Extension qos-bw-limit-direction not supported by any of loaded plugins May 15 10:18:35.771242 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: qos_bw_minimum_ingress.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.771460 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Ingress direction for QoS minimum bandwidth rule" alias="qos-bw-minimum-ingress" description="Allow to configure QoS minumum bandwidth rule with ingress direction." updated="2018-07-09T10:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.771606 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Extension qos-bw-minimum-ingress not supported by any of loaded plugins May 15 10:18:35.771688 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: qos_default.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.771882 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="QoS default policy" alias="qos-default" description="Expose the QoS default policy per project" updated="2017-041-06T10:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.772021 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Extension qos-default not supported by any of loaded plugins May 15 10:18:35.772091 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: qos_fip.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.772315 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Floating IP QoS" alias="qos-fip" description="The floating IP Quality of Service extension" updated="2017-07-20T00:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.772463 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Extension qos-fip not supported by any of loaded plugins May 15 10:18:35.772532 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: qos_fip_network_policy.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.772725 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="QoS floating IP network policy ID" alias="qos-fip-network-policy" description="Adds a the QoS network ID to the floating IP definition" updated="2021-11-15T10:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.772886 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Extension qos-fip-network-policy not supported by any of loaded plugins May 15 10:18:35.772987 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: qos_gateway_ip.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.773186 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Router gateway IP QoS" alias="qos-gateway-ip" description="The Router gateway IP Quality of Service extension" updated="2018-02-24T00:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.773345 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Extension qos-gateway-ip not supported by any of loaded plugins May 15 10:18:35.773421 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: qos_port_network_policy.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.773603 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="QoS port network policy ID" alias="qos-port-network-policy" description="Adds a the QoS network ID to the port definition" updated="2019-11-01T10:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.773752 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Extension qos-port-network-policy not supported by any of loaded plugins May 15 10:18:35.773843 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: qos_pps_minimum_rule.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.774044 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="QoS minimum packet rate rule" alias="qos-pps-minimum" description="Allow to configure QoS minimum packet rate rule." updated="2021-07-14T10:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.774179 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Extension qos-pps-minimum not supported by any of loaded plugins May 15 10:18:35.774250 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: qos_pps_minimum_rule_alias.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.774444 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="QoS minimum packet rate rule alias" alias="qos-pps-minimum-rule-alias" description="API to enable GET, PUT and DELETE operations on QoS minimum packet rate rule without specifying policy ID" updated="2021-10-22T10:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.774574 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Extension qos-pps-minimum-rule-alias not supported by any of loaded plugins May 15 10:18:35.774645 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: qos_pps_rule.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.774857 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="QoS Rule Type Packet per Second Extension" alias="qos-pps" description="Add QoS Rule Type Packet per Second" updated="2021-05-12T10:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.774992 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Extension qos-pps not supported by any of loaded plugins May 15 10:18:35.775062 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: qos_rule_type_details.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.775254 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Details of QoS rule types" alias="qos-rule-type-details" description="Expose details about QoS rule types supported by loaded backend drivers" updated="2017-06-22T10:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.775389 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Extension qos-rule-type-details not supported by any of loaded plugins May 15 10:18:35.775463 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: qos_rule_type_filter.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.775638 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Allow to filter the list of QoS rule types" alias="qos-rule-type-filter" description="Allows to filter the QoS rule type list adding two new flags. "all_rules" prints all implemented QoS rule types."all_supported" prints all supported QoS rule types by the loaded mechanism drivers" updated="2022-02-02T10:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.775788 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Extension qos-rule-type-filter not supported by any of loaded plugins May 15 10:18:35.775858 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: qos_rules_alias.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.776048 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Quality of Service rules alias API" alias="qos-rules-alias" description="API to enable GET, PUT and DELETE operations on QoS policy rules without specifying policy ID" updated="2018-10-07T10:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.776198 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Extension qos-rules-alias not supported by any of loaded plugins May 15 10:18:35.776336 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: quota_check_limit.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.776523 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Quota engine limit check" alias="quota-check-limit" description="Support for checking the resource usage before applying a new quota limit" updated="2021-09-08T16:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.776658 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loaded extension: quota-check-limit May 15 10:18:35.776745 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: quota_check_limit_default.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.776922 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Quota engine limit check by default" alias="quota-check-limit-default" description="By default, the Neutron quota engine checks the resource usage before applying a new quota limit" updated="2024-08-21T16:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.777053 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loaded extension: quota-check-limit-default May 15 10:18:35.777126 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: quotasv2.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.777545 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Quota management support" alias="quotas" description="Expose functions for quotas management per project" updated="2012-07-29T10:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.777621 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loaded extension: quotas May 15 10:18:35.777712 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: quotasv2_detail.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.778219 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Quota details management support" alias="quota_details" description="Expose functions for quotas usage statistics per project" updated="2017-02-10T10:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.778329 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loaded extension: quota_details May 15 10:18:35.778431 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: rbac.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.778942 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="RBAC Policies" alias="rbac-policies" description="Allows creation and modification of policies that control project access to resources." updated="2015-06-17T12:15:12-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.779083 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loaded extension: rbac-policies May 15 10:18:35.779199 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: rbac_address_group.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.779406 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Add address_group type to RBAC" alias="rbac-address-group" description="Add address_group type to network RBAC" updated="2021-01-20T00:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.779555 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Extension rbac-address-group not supported by any of loaded plugins May 15 10:18:35.779613 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: rbac_address_scope.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.779819 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Add address_scope type to RBAC" alias="rbac-address-scope" description="Add address_scope type to RBAC" updated="2020-02-12T00:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.779956 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loaded extension: rbac-address-scope May 15 10:18:35.780044 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: rbac_security_groups.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.780310 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Add security_group type to network RBAC" alias="rbac-security-groups" description="Add security_group type to network RBAC" updated="2019-02-14T00:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.780494 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loaded extension: rbac-security-groups May 15 10:18:35.780593 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: rbac_subnetpool.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.780803 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Add subnetpool type to RBAC" alias="rbac-subnetpool" description="Add subnetpool type to RBAC" updated="2020-02-05T00:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.780948 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Extension rbac-subnetpool not supported by any of loaded plugins May 15 10:18:35.781008 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: revisionifmatch.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.781200 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="If-Match constraints based on revision_number" alias="revision-if-match" description="Extension indicating that If-Match based on revision_number is supported." updated="2016-12-11T00:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.781332 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loaded extension: revision-if-match May 15 10:18:35.781402 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: revisions.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.781615 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Resource revision numbers" alias="standard-attr-revisions" description="This extension will display the revision number of neutron resources." updated="2016-04-11T10:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.781761 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loaded extension: standard-attr-revisions May 15 10:18:35.781837 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: router_availability_zone.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.782065 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Router Availability Zone" alias="router_availability_zone" description="Availability zone support for router." updated="2015-01-01T10:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.782241 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loaded extension: router_availability_zone May 15 10:18:35.782336 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: router_enable_snat.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.782573 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Exposes the router "enable_snat" read-only configuration flag" alias="router-enable-snat" description="Exposes the router "enable_snat" read-only configuration flag" updated="2025-09-18T18:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.782771 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loaded extension: router-enable-snat May 15 10:18:35.782876 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: routerservicetype.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.783117 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Router Service Type" alias="router-service-type" description="Provides router service type" updated="2013-01-29T00:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.783297 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Extension router-service-type not supported by any of loaded plugins May 15 10:18:35.783396 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: security_groups_default_rules.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.783892 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Default rules for security groups" alias="security-groups-default-rules" description="Configure set of security group rules used as default rules for every new security group" updated="2022-12-19T10:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.784079 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loaded extension: security-groups-default-rules May 15 10:18:35.784182 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: security_groups_normalized_cidr.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.784457 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Normalized CIDR field for security group rules" alias="security-groups-normalized-cidr" description="Add new field with normalized remote_ip_prefix cidr in SG rule" updated="2020-07-28T10:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.784646 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loaded extension: security-groups-normalized-cidr May 15 10:18:35.784720 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: security_groups_port_filtering.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.784927 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Port filtering on security groups" alias="port-security-groups-filtering" description="Provides security groups filtering when listing ports" updated="2018-01-09T09:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.785065 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Extension port-security-groups-filtering not supported by any of loaded plugins May 15 10:18:35.785135 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: security_groups_remote_address_group.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.785325 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Remote address group id field for security group rules" alias="security-groups-remote-address-group" description="Add new field of remote address group id in SG rules" updated="2020-08-25T10:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.785457 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loaded extension: security-groups-remote-address-group May 15 10:18:35.785531 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: security_groups_rules_belongs_to_default_sg.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.785742 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Security group rule belongs to the project's default security group" alias="security-groups-rules-belongs-to-default-sg" description="Flag to determine if the security group rule belongs to the project's default security group" updated="2023-05-23T10:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.785871 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loaded extension: security-groups-rules-belongs-to-default-sg May 15 10:18:35.785942 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: security_groups_shared_filtering.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.786141 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Security group filtering on the shared field" alias="security-groups-shared-filtering" description="Support filtering security groups on the shared field" updated="2021-10-05T09:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.786307 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loaded extension: security-groups-shared-filtering May 15 10:18:35.786403 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: securitygroup.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.788323 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="security-group" alias="security-group" description="The security groups extension." updated="2012-10-05T10:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.788525 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loaded extension: security-group May 15 10:18:35.788626 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: segment.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.788999 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Segment" alias="segment" description="Segments extension." updated="2016-02-24T17:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.789185 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Extension segment not supported by any of loaded plugins May 15 10:18:35.789263 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: segments_peer_subnet_host_routes.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.789464 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Segments peer-subnet host routes" alias="segments-peer-subnet-host-routes" description="Add host routes to subnets on a routed network (segments)" updated="2018-06-12T10:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.789602 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Extension segments-peer-subnet-host-routes not supported by any of loaded plugins May 15 10:18:35.789670 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: servicetype.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.789896 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Neutron Service Type Management" alias="service-type" description="API for retrieving service providers for Neutron advanced services" updated="2013-01-20T00:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.790026 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loaded extension: service-type May 15 10:18:35.790097 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: sorting.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.790298 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Sorting support" alias="sorting" description="Extension that indicates that sorting is enabled." updated="2016-06-12T00:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.790370 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loaded extension: sorting May 15 10:18:35.790440 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: standard_attr_segment.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.790619 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Standard Attribute Segment Extension" alias="standard-attr-segment" description="Add standard attributes to Segment resource" updated="2018-04-09T10:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.790776 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Extension standard-attr-segment not supported by any of loaded plugins May 15 10:18:35.790847 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: standardattrdescription.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.791059 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="standard-attr-description" alias="standard-attr-description" description="Extension to add descriptions to standard attributes" updated="2016-02-10T10:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.791232 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loaded extension: standard-attr-description May 15 10:18:35.791339 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: stateful_security_group.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.791591 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Stateful security group" alias="stateful-security-group" description="Indicates if the security group is stateful or not" updated="2019-11-26T09:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.791789 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loaded extension: stateful-security-group May 15 10:18:35.791881 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: stdattrs_common.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.792099 np0000174319 neutron-ovn-maintenance-worker[104616]: WARNING neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Did not find expected name "Stdattrs_common" in /opt/stack/neutron/neutron/extensions/stdattrs_common.py May 15 10:18:35.792197 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: subnet_dns_publish_fixed_ip.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.792490 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Subnet DNS publish fixed IP" alias="subnet-dns-publish-fixed-ip" description="Support choosing to publish DNS records for IPs from a subnet" updated="2019-05-24T10:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.792669 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loaded extension: subnet-dns-publish-fixed-ip May 15 10:18:35.792783 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: subnet_external_network.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.793011 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Subnet belongs to an external network" alias="subnet-external-network" description="Informs if the subnet belongs to an external network" updated="2024-02-05T18:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.793188 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loaded extension: subnet-external-network May 15 10:18:35.793284 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: subnet_onboard.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.793533 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Subnet Onboard" alias="subnet_onboard" description="Provides support for onboarding subnets into subnet pools" updated="2018-12-18T09:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.793725 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Extension subnet_onboard not supported by any of loaded plugins May 15 10:18:35.793821 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: subnet_segmentid_writable.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.794045 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Subnet SegmentID (writable)" alias="subnet-segmentid-writable" description="Provides a writable segment_id attribute for a subnet resource." updated="2018-03-12T00:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.794225 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Extension subnet-segmentid-writable not supported by any of loaded plugins May 15 10:18:35.794322 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: subnet_service_types.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.794697 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Subnet service types" alias="subnet-service-types" description="Provides ability to set the subnet service_types field" updated="2016-03-15T18:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.795013 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loaded extension: subnet-service-types May 15 10:18:35.795162 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: subnetallocation.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.795802 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Subnet Allocation" alias="subnet_allocation" description="Enables allocation of subnets from a subnet pool" updated="2015-03-30T10:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.796102 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loaded extension: subnet_allocation May 15 10:18:35.796328 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: subnetpool_prefix_ops.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.796632 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Subnet Pool Prefix Operations" alias="subnetpool-prefix-ops" description="Provides support for adjusting the prefix list of subnet pools" updated="2019-02-08T10:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.796826 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loaded extension: subnetpool-prefix-ops May 15 10:18:35.796919 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: tag_creation.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.797206 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Tag creation extension" alias="tag-creation" description="Allow to create multiple tags for a resource" updated="2024-23-07T10:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.797379 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loaded extension: tag-creation May 15 10:18:35.797496 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: tag_ports_during_bulk_creation.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.797899 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Tag Ports During Bulk Creation" alias="tag-ports-during-bulk-creation" description="Allow to tag ports during bulk creation" updated="2019-12-29T19:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.798127 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Extension tag-ports-during-bulk-creation not supported by any of loaded plugins May 15 10:18:35.798221 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: tagging.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.799170 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Tag support for resources with standard attribute: port, subnet, subnetpool, network, security_group, router, floatingip, policy, trunk, network_segment_range" alias="standard-attr-tag" description="Enables to set tag on resources with standard attribute." updated="2017-01-01T00:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.799328 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loaded extension: standard-attr-tag May 15 10:18:35.799399 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: timestamp.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.799624 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Resource timestamps" alias="standard-attr-timestamp" description="Adds created_at and updated_at fields to all Neutron resources that have Neutron standard attributes." updated="2016-09-12T10:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.799781 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loaded extension: standard-attr-timestamp May 15 10:18:35.799877 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: trunk.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.800122 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Trunk Extension" alias="trunk" description="Provides support for trunk ports" updated="2016-01-01T10:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.800336 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Extension trunk not supported by any of loaded plugins May 15 10:18:35.800427 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: trunk_details.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.800663 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Trunk port details" alias="trunk-details" description="Expose trunk port details" updated="2016-01-01T10:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.800854 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Extension trunk-details not supported by any of loaded plugins May 15 10:18:35.800951 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: uplink_status_propagation.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.801195 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Uplink status propagation extension" alias="uplink-status-propagation" description="Provides ability to propagate port uplink status." updated="2018-05-31T18:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.801365 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Extension uplink-status-propagation not supported by any of loaded plugins May 15 10:18:35.801461 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: uplink_status_propagation_updatable.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.801689 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Uplink status propagation updatable extension" alias="uplink-status-propagation-updatable" description="Provides ability to update the port uplink status." updated="2024-09-03T18:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.801883 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Extension uplink-status-propagation-updatable not supported by any of loaded plugins May 15 10:18:35.801961 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loading extension file: vlantransparent.py {{(pid=104616) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 15 10:18:35.802233 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Ext name="Vlantransparent" alias="vlan-transparent" description="Provides Vlan Transparent Networks" updated="2015-03-23T09:00:00-00:00" {{(pid=104616) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 15 10:18:35.802410 np0000174319 neutron-ovn-maintenance-worker[104616]: INFO neutron.api.extensions [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Loaded extension: vlan-transparent May 15 10:18:35.805253 np0000174319 neutron-ovn-maintenance-worker[104616]: WARNING oslo_service.backend._threading.service [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Service MaintenanceWorker is not picklable with spawn; falling back to fork. Please make the service spawn-safe to avoid this fallback. May 15 10:18:35.805637 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Full set of CONF: {{(pid=104616) _load_service_manager_options /opt/stack/data/venv/lib/python3.12/site-packages/cotyledon/oslo_config_glue.py:59}} May 15 10:18:35.805719 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] ******************************************************************************** {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2979}} May 15 10:18:35.805774 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Configuration options gathered from: {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2980}} May 15 10:18:35.805856 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] command line args: ['--config-file', '/etc/neutron/neutron.conf', '--config-file', '/etc/neutron/plugins/ml2/ml2_conf.ini'] {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2981}} May 15 10:18:35.805957 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] config files: ['/etc/neutron/neutron.conf', '/etc/neutron/plugins/ml2/ml2_conf.ini'] {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2982}} May 15 10:18:35.806032 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] ================================================================================ {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2987}} May 15 10:18:35.806128 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] agent_down_time = 75 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.806216 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] allow_automatic_dhcp_failover = True {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.806302 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] allow_bulk = True {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.806477 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] allowed_conntrack_helpers = [{'amanda': 'tcp'}, {'ftp': 'tcp'}, {'h323': 'udp'}, {'h323': 'tcp'}, {'irc': 'tcp'}, {'netbios-ns': 'udp'}, {'pptp': 'tcp'}, {'sane': 'tcp'}, {'sip': 'udp'}, {'sip': 'tcp'}, {'snmp': 'udp'}, {'tftp': 'udp'}] {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.806536 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] api_extensions_path = {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.806665 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] api_paste_config = api-paste.ini {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.806798 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] api_workers = 4 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.806896 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] auth_strategy = keystone {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.806988 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] base_mac = fa:16:3e:00:00:00 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.807108 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] bind_host = 0.0.0.0 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.807245 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] bind_port = 9696 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.807352 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] client_socket_timeout = 900 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.807481 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] config_dir = [] {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.807576 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] config_file = ['/etc/neutron/neutron.conf', '/etc/neutron/plugins/ml2/ml2_conf.ini'] {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.807713 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] config_source = [] {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.807835 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] control_exchange = neutron {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.807953 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] core_plugin = ml2 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.808071 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] debug = True {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.808185 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] default_availability_zones = [] {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.808436 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] default_log_levels = ['amqp=WARN', 'boto=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'OFPHandler=INFO', 'OfctlService=INFO', 'os_ken.base.app_manager=INFO', 'os_ken.controller.controller=INFO'] {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.808546 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] dhcp_agent_notification = False {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.808659 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] dhcp_agents_per_network = 1 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.808773 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] dhcp_lease_duration = 86400 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.808871 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] dhcp_load_type = networks {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.808960 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] dns_domain = openstacklocal {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.809047 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] dvr_base_mac = fa:16:3f:00:00:00 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.809133 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] enable_default_route_bfd = False {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.809214 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] enable_default_route_ecmp = False {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.809299 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] enable_dvr = True {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.809385 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] enable_new_agents = True {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.809469 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] enable_services_on_agents_with_admin_state_down = False {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.809552 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] enable_snat_by_default = True {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.809636 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] enable_traditional_dhcp = True {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.809725 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] external_dns_driver = None {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.809816 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] filter_validation = True {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.809906 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] global_physnet_mtu = 1430 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.809959 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] graceful_shutdown_timeout = 60 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.810097 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] host = np0000174319 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.810188 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] host_dvr_for_dhcp = True {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.810275 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] http_retries = 3 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.810362 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] instance_format = [instance: %(uuid)s] {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.810452 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] instance_uuid_format = [instance: %(uuid)s] {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.810535 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] ipam_driver = internal {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.810620 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] ipv6_pd_enabled = False {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.810713 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] l3_ha = False {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.810806 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] l3_ha_net_cidr = 169.254.192.0/18 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.810892 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] l3_ha_network_physical_name = {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.810978 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] l3_ha_network_type = {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.811066 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] log_color = False {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.811177 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] log_config_append = None {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.811278 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.811365 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] log_dir = None {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.811450 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] log_file = None {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.811504 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] log_options = True {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.811598 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] log_rotate_interval = 1 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.811694 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] log_rotate_interval_type = days {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.811823 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] log_rotation_type = none {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.811895 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.811964 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.812055 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.812144 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.812198 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.812355 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] max_allowed_address_pair = 10 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.812442 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] max_dns_nameservers = 5 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.812527 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] max_header_line = 16384 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.812610 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] max_l3_agents_per_router = 3 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.812713 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] max_logfile_count = 30 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.812799 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] max_logfile_size_mb = 200 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.812884 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] max_routes = 30 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.812968 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] max_subnet_host_routes = 20 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.813056 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] my_ip = 199.204.45.132 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.813135 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] my_ipv6 = 2604:e100:1:0:f816:3eff:fe62:7392 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.813226 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] network_auto_schedule = True {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.813300 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] network_link_prefix = None {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.813400 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] network_scheduler_driver = neutron.scheduler.dhcp_agent_scheduler.WeightScheduler {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.813495 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] notify_nova_on_port_data_changes = True {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.813582 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] notify_nova_on_port_status_changes = True {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.813670 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] pagination_max_limit = -1 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.813770 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] periodic_fuzzy_delay = 5 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.813841 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] periodic_interval = 40 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.813935 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] publish_errors = False {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.813999 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] rate_limit_burst = 0 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.814102 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] rate_limit_except_level = CRITICAL {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.814184 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] rate_limit_interval = 0 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.814268 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] router_distributed = False {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.814357 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] rpc_resources_processing_step = 20 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.814442 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] rpc_response_max_timeout = 600 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.814527 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] rpc_response_timeout = 60 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.814615 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] rpc_state_report_workers = 0 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.814710 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] rpc_workers = None {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.814796 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] send_events_interval = 2 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.814890 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] service_plugins = ['ovn-router'] {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.815004 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] setproctitle = on {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.815131 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] shell_completion = None {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.815244 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] state_path = /opt/stack/data/neutron {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.815341 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] syslog_log_facility = LOG_USER {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.815440 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] tcp_keepidle = 600 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.815506 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] transport_url = **** {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.815599 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] use_journal = False {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.815702 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] use_json = False {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.815805 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] use_stderr = False {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.815918 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] use_syslog = False {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.816036 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] vlan_qinq = None {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.816147 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] vlan_transparent = None {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.816270 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] wsgi_default_pool_size = 100 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.816341 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] wsgi_keep_alive = True {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.816436 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] wsgi_log_format = %(client_ip)s "%(request_line)s" status: %(status_code)s len: %(body_length)s time: %(wall_seconds).7f {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.816521 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] wsgi_server_debug = False {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.816625 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] oslo_concurrency.disable_process_locking = False {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.816739 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] oslo_concurrency.lock_path = /opt/stack/data/neutron/lock {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.816840 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] profiler.connection_string = messaging:// {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.816928 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] profiler.enabled = False {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.817019 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] profiler.es_doc_type = notification {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.817108 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] profiler.es_scroll_size = 10000 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.817198 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] profiler.es_scroll_time = 2m {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.817285 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] profiler.filter_error_trace = False {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.817373 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] profiler.hmac_keys = **** {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.817463 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] profiler.sentinel_service_name = mymaster {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.817560 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] profiler.socket_timeout = 0.1 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.817647 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] profiler.trace_requests = False {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.817800 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] profiler.trace_sqlalchemy = False {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.817941 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] profiler_jaeger.process_tags = {} {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.818067 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] profiler_jaeger.service_name_prefix = None {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.818161 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] profiler_otlp.service_name_prefix = None {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.818254 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.818340 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] oslo_messaging_metrics.metrics_enabled = False {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.818431 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] oslo_messaging_metrics.metrics_process_name = {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.818519 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.818605 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.818725 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] oslo_policy.enforce_new_defaults = True {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.818916 np0000174319 neutron-ovn-maintenance-worker[104616]: WARNING oslo_config.cfg [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Deprecated: Option "enforce_scope" from group "oslo_policy" is deprecated for removal (This configuration was added temporarily to facilitate a smooth transition to the new RBAC. OpenStack will always enforce scope checks. This configuration option is deprecated and will be removed in the 2025.2 cycle.). Its value may be silently ignored in the future. May 15 10:18:35.819011 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] oslo_policy.enforce_scope = True {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.819109 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] oslo_policy.policy_default_rule = default {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.819252 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] oslo_policy.policy_dirs = ['policy.d'] {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.819391 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] oslo_policy.policy_file = /etc/neutron/policy.json {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.819527 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.819664 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.819804 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] oslo_policy.remote_ssl_client_crt_file = None {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.819919 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] oslo_policy.remote_ssl_client_key_file = None {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.820028 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.820142 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] oslo_policy.remote_timeout = 60.0 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.820331 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.820448 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] service_providers.service_provider = [] {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.820567 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] nova.auth_section = None {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.820695 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] nova.auth_type = password {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.820790 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] nova.auth_url = https://199.204.45.132/identity {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.820887 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] nova.cafile = None {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.820977 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] nova.certfile = None {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.821075 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] nova.collect_timing = False {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.821169 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] nova.default_domain_id = None {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.821262 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] nova.default_domain_name = None {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.821364 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] nova.domain_id = None {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.821453 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] nova.domain_name = None {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.821567 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] nova.endpoint_type = public {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.821692 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] nova.insecure = False {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.821801 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] nova.keyfile = None {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.821919 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] nova.password = **** {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.822037 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] nova.project_domain_id = None {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.822143 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] nova.project_domain_name = Default {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.822256 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] nova.project_id = None {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.822371 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] nova.project_name = service {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.822484 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] nova.region_name = RegionOne {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.822594 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] nova.split_loggers = False {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.822717 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] nova.system_scope = None {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.822823 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] nova.timeout = None {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.822929 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] nova.trust_id = None {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.823035 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] nova.user_domain_id = None {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.823150 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] nova.user_domain_name = Default {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.823257 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] nova.user_id = None {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.823380 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] nova.username = nova {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.823499 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] placement.auth_section = None {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.823625 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] placement.auth_type = password {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.823770 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] placement.cafile = None {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.823889 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] placement.certfile = None {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.824006 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] placement.collect_timing = False {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.824136 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] placement.endpoint_type = public {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.824280 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] placement.insecure = False {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.824394 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] placement.keyfile = None {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.824512 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] placement.region_name = None {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.824635 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] placement.split_loggers = False {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.824769 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] placement.timeout = None {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.824891 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] ironic.auth_section = None {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.825001 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] ironic.auth_type = None {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.825116 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] ironic.cafile = None {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.825233 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] ironic.certfile = None {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.825360 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] ironic.collect_timing = False {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.825482 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] ironic.connect_retries = None {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.825609 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] ironic.connect_retry_delay = None {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.825755 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] ironic.enable_notifications = False {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.825880 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] ironic.endpoint_override = None {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.826006 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] ironic.insecure = False {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.826126 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] ironic.interface = None {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.826245 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] ironic.keyfile = None {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.826367 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] ironic.max_version = None {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.826484 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] ironic.min_version = None {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.826602 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] ironic.region_name = None {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.826740 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] ironic.retriable_status_codes = None {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.826853 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] ironic.service_name = None {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.826972 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] ironic.service_type = None {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.827097 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] ironic.split_loggers = False {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.827224 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] ironic.status_code_retries = None {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.827350 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] ironic.status_code_retry_delay = None {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.827472 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] ironic.timeout = None {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.827591 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] ironic.valid_interfaces = None {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.827730 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] ironic.version = None {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.827859 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] cli_script.dry_run = False {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.827993 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] oslo_messaging_rabbit.amqp_auto_delete = False {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.828113 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] oslo_messaging_rabbit.amqp_durable_queues = False {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.828284 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] oslo_messaging_rabbit.conn_pool_min_size = 2 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.828398 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] oslo_messaging_rabbit.conn_pool_ttl = 1200 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.828517 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] oslo_messaging_rabbit.direct_mandatory_flag = True {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.828636 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] oslo_messaging_rabbit.enable_cancel_on_failover = False {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.828776 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] oslo_messaging_rabbit.heartbeat_in_pthread = False {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.828898 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] oslo_messaging_rabbit.heartbeat_rate = 3 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.829017 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] oslo_messaging_rabbit.heartbeat_timeout_threshold = 60 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.829149 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] oslo_messaging_rabbit.hostname = np0000174319 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.829265 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] oslo_messaging_rabbit.kombu_compression = None {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.829390 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] oslo_messaging_rabbit.kombu_failover_strategy = round-robin {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.829515 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] oslo_messaging_rabbit.kombu_missing_consumer_retry_timeout = 60 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.829640 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] oslo_messaging_rabbit.kombu_reconnect_delay = 1.0 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.829784 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] oslo_messaging_rabbit.kombu_reconnect_splay = 0.0 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.829914 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] oslo_messaging_rabbit.processname = neutron-ovn-maintenance-worker {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.830033 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] oslo_messaging_rabbit.rabbit_ha_queues = False {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.830154 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] oslo_messaging_rabbit.rabbit_interval_max = 30 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.830280 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] oslo_messaging_rabbit.rabbit_login_method = AMQPLAIN {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.830399 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] oslo_messaging_rabbit.rabbit_qos_prefetch_count = 0 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.830520 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] oslo_messaging_rabbit.rabbit_quorum_delivery_limit = 0 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.830637 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] oslo_messaging_rabbit.rabbit_quorum_max_memory_bytes = 0 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.830774 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] oslo_messaging_rabbit.rabbit_quorum_max_memory_length = 0 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.830893 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] oslo_messaging_rabbit.rabbit_quorum_queue = False {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.831010 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] oslo_messaging_rabbit.rabbit_retry_backoff = 2 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.831128 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] oslo_messaging_rabbit.rabbit_retry_interval = 1 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.831249 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] oslo_messaging_rabbit.rabbit_stream_fanout = False {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.831367 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] oslo_messaging_rabbit.rabbit_transient_queues_ttl = 1800 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.831484 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] oslo_messaging_rabbit.rabbit_transient_quorum_queue = False {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.831602 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] oslo_messaging_rabbit.rpc_conn_pool_size = 30 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.831763 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] oslo_messaging_rabbit.ssl = False {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.831886 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] oslo_messaging_rabbit.ssl_ca_file = {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.831976 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] oslo_messaging_rabbit.ssl_cert_file = {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.832089 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] oslo_messaging_rabbit.ssl_enforce_fips_mode = False {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.832267 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] oslo_messaging_rabbit.ssl_key_file = {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.832389 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] oslo_messaging_rabbit.ssl_version = {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.832508 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] oslo_messaging_rabbit.use_queue_manager = False {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.832651 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] oslo_messaging_notifications.driver = [] {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.832772 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] oslo_messaging_notifications.retry = -1 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.832907 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] oslo_messaging_notifications.topics = ['notifications'] {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.833028 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] oslo_messaging_notifications.transport_url = **** {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.833153 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] oslo_reports.file_event_handler = None {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.833273 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] oslo_reports.file_event_handler_interval = 1 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.833388 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] oslo_reports.log_dir = None {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.833604 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] oslo_versionedobjects.fatal_exception_format_errors = False {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.833836 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] QUOTAS.default_quota = -1 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.834016 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] QUOTAS.quota_driver = neutron.db.quota.driver_nolock.DbQuotaNoLockDriver {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.834215 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] QUOTAS.quota_floatingip = 50 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.834340 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] QUOTAS.quota_network = 100 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.834452 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] QUOTAS.quota_port = 500 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.834566 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] QUOTAS.quota_rbac_policy = 10 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.834690 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] QUOTAS.quota_router = 10 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.834813 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] QUOTAS.quota_security_group = 10 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.834928 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] QUOTAS.quota_security_group_rule = 100 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.835037 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] QUOTAS.quota_subnet = 100 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.835150 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] QUOTAS.track_quota_usage = True {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.835273 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] privsep.capabilities = [21, 12, 1, 2, 19] {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.835383 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] privsep.group = None {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.835490 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] privsep.helper_command = None {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.835599 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] privsep.log_daemon_traceback = False {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.835731 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] privsep.logger_name = oslo_privsep.daemon {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.835836 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] privsep.thread_pool_size = 16 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.835948 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] privsep.user = None {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.836294 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] privsep_dhcp_release.capabilities = [21, 12] {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.836409 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] privsep_dhcp_release.group = None {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.836520 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] privsep_dhcp_release.helper_command = None {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.836629 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] privsep_dhcp_release.log_daemon_traceback = False {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.836809 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] privsep_dhcp_release.logger_name = oslo_privsep.daemon {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.836917 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] privsep_dhcp_release.thread_pool_size = 16 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.837023 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] privsep_dhcp_release.user = None {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.837144 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] privsep_ovs_vsctl.capabilities = [21, 12] {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.837253 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] privsep_ovs_vsctl.group = None {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.837361 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] privsep_ovs_vsctl.helper_command = None {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.837479 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] privsep_ovs_vsctl.log_daemon_traceback = False {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.837584 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] privsep_ovs_vsctl.logger_name = oslo_privsep.daemon {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.837732 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] privsep_ovs_vsctl.thread_pool_size = 16 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.837886 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] privsep_ovs_vsctl.user = None {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.838004 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] privsep_namespace.capabilities = [21] {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.838254 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] privsep_namespace.group = None {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.838327 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] privsep_namespace.helper_command = None {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.838438 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] privsep_namespace.log_daemon_traceback = False {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.838547 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] privsep_namespace.logger_name = oslo_privsep.daemon {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.838653 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] privsep_namespace.thread_pool_size = 16 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.838782 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] privsep_namespace.user = None {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.838900 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] privsep_conntrack.capabilities = [12] {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.838995 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] privsep_conntrack.group = None {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.839098 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] privsep_conntrack.helper_command = None {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.839205 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] privsep_conntrack.log_daemon_traceback = False {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.839317 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] privsep_conntrack.logger_name = oslo_privsep.daemon {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.839423 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] privsep_conntrack.thread_pool_size = 16 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.839527 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] privsep_conntrack.user = None {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.839642 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] privsep_link.capabilities = [12, 21] {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.839765 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] privsep_link.group = None {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.839862 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] privsep_link.helper_command = None {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.840161 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] privsep_link.log_daemon_traceback = False {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.840278 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] privsep_link.logger_name = oslo_privsep.daemon {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.840394 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] privsep_link.thread_pool_size = 16 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.840489 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] privsep_link.user = None {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.840617 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] AGENT.root_helper = sudo /opt/stack/data/venv/bin/neutron-rootwrap /etc/neutron/rootwrap.conf {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.840746 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] AGENT.root_helper_daemon = sudo /opt/stack/data/venv/bin/neutron-rootwrap-daemon /etc/neutron/rootwrap.conf {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.840870 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] AGENT.use_helper_for_ns_read = True {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.840981 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] SECURITYGROUP.enable_ipset = True {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.841083 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] SECURITYGROUP.enable_security_group = True {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.841286 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] SECURITYGROUP.firewall_driver = None {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.841286 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] SECURITYGROUP.permitted_ethertypes = [] {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.841394 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] experimental.ipv6_pd_enabled = False {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.841545 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] ml2.extension_drivers = ['port_security', 'qos'] {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.841635 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] ml2.external_network_type = None {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.841770 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] ml2.mechanism_drivers = ['ovn'] {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.841871 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] ml2.overlay_ip_version = 4 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.842057 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] ml2.path_mtu = 0 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.842147 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] ml2.physical_network_mtus = {} {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.842207 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] ml2.project_network_types = ['geneve'] {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.842340 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] ml2.tunnelled_network_rp_name = rp_tunnelled {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.842458 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] ml2.type_drivers = ['local', 'flat', 'vlan', 'geneve'] {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.842587 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] ml2_type_flat.flat_networks = ['public'] {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.842727 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] ml2_type_geneve.max_header_size = 38 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.842843 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] ml2_type_geneve.vni_ranges = ['1:65536'] {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.842968 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] ml2_type_vlan.network_vlan_ranges = ['public'] {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.843068 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] network_log.burst_limit = 25 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.843189 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] network_log.local_output_log_base = None {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.843297 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] network_log.rate_limit = 100 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.843411 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] database.asyncio_connection = **** {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.843516 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] database.asyncio_slave_connection = **** {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.843628 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] database.backend = sqlalchemy {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.843757 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] database.connection = **** {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.843868 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] database.connection_debug = 0 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.843984 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] database.connection_parameters = {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.844087 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] database.connection_recycle_time = 3600 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.844264 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] database.connection_trace = False {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.844354 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] database.db_inc_retry_interval = True {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.844467 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] database.db_max_retries = 20 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.844575 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] database.db_max_retry_interval = 10 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.844700 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] database.db_retry_interval = 1 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.844810 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] database.max_overflow = 50 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.844914 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] database.max_pool_size = 5 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.845022 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] database.max_retries = 10 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.845134 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] database.mysql_sql_mode = TRADITIONAL {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.845240 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] database.mysql_wsrep_sync_wait = None {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.845345 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] database.pool_timeout = None {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.845454 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] database.retry_interval = 10 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.845559 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] database.slave_connection = **** {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.845667 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] database.sqlite_synchronous = True {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.845804 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] database.synchronous_reader = True {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.845910 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] database.use_db_reconnect = False {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.846034 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] ovn.bfd_min_rx = 1000 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.846140 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] ovn.bfd_min_tx = 100 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.846248 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] ovn.bfd_mult = 3 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.846355 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] ovn.broadcast_arps_to_all_routers = True {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.846464 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] ovn.dhcp_default_lease_time = 43200 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.846572 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] ovn.disable_ovn_dhcp_for_baremetal_ports = False {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.846691 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] ovn.dns_records_ovn_owned = False {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.846814 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] ovn.dns_servers = [] {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.846925 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] ovn.enable_distributed_floating_ip = False {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.847036 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] ovn.fdb_age_threshold = 300 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.847150 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] ovn.ha_failover_strategy = normal {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.847273 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] ovn.live_migration_activation_strategy = rarp {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.847383 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] ovn.localnet_learn_fdb = False {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.847493 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] ovn.mac_binding_age_threshold = 300 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.847609 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] ovn.neutron_sync_mode = log {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.847752 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] ovn.ovn_dhcp4_global_options = {} {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.847855 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] ovn.ovn_dhcp6_global_options = {} {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.847984 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] ovn.ovn_l3_scheduler = leastloaded {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.848089 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] ovn.ovn_metadata_enabled = True {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.848234 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] ovn.ovn_nb_ca_cert = /opt/stack/data/CA/int-ca/ca-chain.pem {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.848351 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] ovn.ovn_nb_certificate = /opt/stack/data/CA/int-ca/devstack-cert.crt {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.848473 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] ovn.ovn_nb_connection = ['ssl:199.204.45.132:6641'] {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.848613 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] ovn.ovn_nb_private_key = /opt/stack/data/CA/int-ca/private/devstack-cert.key {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.848711 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] ovn.ovn_router_indirect_snat = False {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.848825 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] ovn.ovn_sb_ca_cert = /opt/stack/data/CA/int-ca/ca-chain.pem {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.848937 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] ovn.ovn_sb_certificate = /opt/stack/data/CA/int-ca/devstack-cert.crt {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.849064 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] ovn.ovn_sb_connection = ['ssl:199.204.45.132:6642'] {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.849168 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] ovn.ovn_sb_private_key = /opt/stack/data/CA/int-ca/private/devstack-cert.key {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.849277 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] ovn.ovs_create_tap = False {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.849387 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] ovn.ovsdb_connection_timeout = 180 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.849503 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] ovn.ovsdb_log_level = INFO {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.849612 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] ovn.ovsdb_probe_interval = 60000 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.849741 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] ovn.ovsdb_retry_max_interval = 180 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.849853 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] ovn.stateless_nat_enabled = False {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.849967 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] ovn.vhost_sock_dir = /var/run/openvswitch {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.850093 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] OVS.bridge_mac_table_size = 50000 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.850197 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] OVS.igmp_flood = False {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.850307 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] OVS.igmp_flood_reports = True {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.850417 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] OVS.igmp_flood_unregistered = False {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.850533 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] OVS.igmp_snooping_enable = False {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.850642 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] OVS.ovsdb_timeout = 10 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.850776 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] ovn_nb_global.fdb_removal_limit = 0 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.850892 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] ovn_nb_global.ignore_lsp_down = False {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.851005 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] ovn_nb_global.mac_binding_removal_limit = 0 {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.851098 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG cotyledon.oslo_config_glue [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] ******************************************************************************** {{(pid=104616) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3013}} May 15 10:18:35.851225 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG oslo_service.backend._threading.service [None req-93713eaa-c3dd-4fad-94d5-d95a7e8659b6 None None] Selected the multiprocessing context: and updated it in Cotyledon ServiceManager: {{(pid=104616) launch_service /opt/stack/data/venv/lib/python3.12/site-packages/oslo_service/backend/_threading/service.py:394}} May 15 10:18:35.875211 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] Full set of CONF: {{(pid=105458) _load_service_options /opt/stack/data/venv/lib/python3.12/site-packages/cotyledon/oslo_config_glue.py:65}} May 15 10:18:35.875743 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] ******************************************************************************** {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2979}} May 15 10:18:35.875743 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] Configuration options gathered from: {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2980}} May 15 10:18:35.875743 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] command line args: ['--config-file', '/etc/neutron/neutron.conf', '--config-file', '/etc/neutron/plugins/ml2/ml2_conf.ini'] {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2981}} May 15 10:18:35.875743 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] config files: ['/etc/neutron/neutron.conf', '/etc/neutron/plugins/ml2/ml2_conf.ini'] {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2982}} May 15 10:18:35.875743 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] ================================================================================ {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2987}} May 15 10:18:35.875743 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] agent_down_time = 75 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.875980 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] allow_automatic_dhcp_failover = True {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.875980 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] allow_bulk = True {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.876116 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] allowed_conntrack_helpers = [{'amanda': 'tcp'}, {'ftp': 'tcp'}, {'h323': 'udp'}, {'h323': 'tcp'}, {'irc': 'tcp'}, {'netbios-ns': 'udp'}, {'pptp': 'tcp'}, {'sane': 'tcp'}, {'sip': 'udp'}, {'sip': 'tcp'}, {'snmp': 'udp'}, {'tftp': 'udp'}] {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.876318 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] api_extensions_path = {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.876441 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] api_paste_config = api-paste.ini {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.876600 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] api_workers = 4 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.876852 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] auth_strategy = keystone {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.876973 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] base_mac = fa:16:3e:00:00:00 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.877197 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] bind_host = 0.0.0.0 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.877327 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] bind_port = 9696 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.877409 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] client_socket_timeout = 900 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.877537 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] config_dir = [] {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.877616 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] config_file = ['/etc/neutron/neutron.conf', '/etc/neutron/plugins/ml2/ml2_conf.ini'] {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.877764 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] config_source = [] {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.877980 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] control_exchange = neutron {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.878093 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] core_plugin = ml2 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.878211 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] debug = True {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.878327 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] default_availability_zones = [] {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.878564 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] default_log_levels = ['amqp=WARN', 'boto=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'OFPHandler=INFO', 'OfctlService=INFO', 'os_ken.base.app_manager=INFO', 'os_ken.controller.controller=INFO'] {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.878689 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] dhcp_agent_notification = False {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.878739 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] dhcp_agents_per_network = 1 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.878852 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] dhcp_lease_duration = 86400 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.878956 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] dhcp_load_type = networks {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.879109 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] dns_domain = openstacklocal {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.879227 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] dvr_base_mac = fa:16:3f:00:00:00 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.879342 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] enable_default_route_bfd = False {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.879423 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] enable_default_route_ecmp = False {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.879524 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] enable_dvr = True {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.879616 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] enable_new_agents = True {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.879740 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] enable_services_on_agents_with_admin_state_down = False {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.879820 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] enable_snat_by_default = True {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.879917 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] enable_traditional_dhcp = True {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.879997 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] external_dns_driver = None {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.880059 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] fatal_deprecations = False {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.880251 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] filter_validation = True {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.880385 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] global_physnet_mtu = 1430 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.880458 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] graceful_shutdown_timeout = 60 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.880744 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] host = np0000174319 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.880832 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] host_dvr_for_dhcp = True {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.880937 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] http_retries = 3 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.881023 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] instance_format = [instance: %(uuid)s] {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.881105 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.881201 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] ipam_driver = internal {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.881293 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] ipv6_pd_enabled = False {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.881507 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] l3_ha = False {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.881620 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] l3_ha_net_cidr = 169.254.192.0/18 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.881746 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] l3_ha_network_physical_name = {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.881825 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] l3_ha_network_type = {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.881934 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] log_color = False {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.882037 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] log_config_append = None {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.882119 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.882209 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] log_dir = None {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.882296 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] log_file = None {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.882345 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] log_options = True {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.882443 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] log_rotate_interval = 1 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.882542 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] log_rotate_interval_type = days {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.882636 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] log_rotation_type = none {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.882740 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.882803 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.882846 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.882942 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.882996 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.883087 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] max_allowed_address_pair = 10 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.883180 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] max_dns_nameservers = 5 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.883271 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] max_header_line = 16384 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.883356 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] max_l3_agents_per_router = 3 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.883446 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] max_logfile_count = 30 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.883527 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] max_logfile_size_mb = 200 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.883625 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] max_routes = 30 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.883739 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] max_subnet_host_routes = 20 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.883830 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] my_ip = 199.204.45.132 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.883926 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] my_ipv6 = 2604:e100:1:0:f816:3eff:fe62:7392 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.884010 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] network_auto_schedule = True {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.884100 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] network_link_prefix = None {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.884237 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] network_scheduler_driver = neutron.scheduler.dhcp_agent_scheduler.WeightScheduler {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.884326 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] notify_nova_on_port_data_changes = True {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.884462 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] notify_nova_on_port_status_changes = True {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.884547 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] pagination_max_limit = -1 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.884639 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] periodic_fuzzy_delay = 5 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.884760 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] periodic_interval = 40 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.884839 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] publish_errors = False {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.884920 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] rate_limit_burst = 0 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.885019 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] rate_limit_except_level = CRITICAL {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.885096 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] rate_limit_interval = 0 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.885182 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] router_distributed = False {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.885270 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] rpc_resources_processing_step = 20 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.885361 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] rpc_response_max_timeout = 600 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.885445 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] rpc_response_timeout = 60 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.885535 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] rpc_state_report_workers = 0 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.885618 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] rpc_workers = None {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.885729 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] send_events_interval = 2 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.885868 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] service_plugins = ['ovn-router'] {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.885961 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] setproctitle = on {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.886059 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] shell_completion = None {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.886156 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] state_path = /opt/stack/data/neutron {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.886262 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] syslog_log_facility = LOG_USER {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.886378 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] tcp_keepidle = 600 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.886445 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] transport_url = **** {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.886530 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] use_journal = False {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.886608 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] use_json = False {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.886729 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] use_stderr = False {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.886808 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] use_syslog = False {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.886884 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] vlan_qinq = None {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.886958 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] vlan_transparent = None {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.887054 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] wsgi_default_pool_size = 100 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.887131 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] wsgi_keep_alive = True {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.887244 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] wsgi_log_format = %(client_ip)s "%(request_line)s" status: %(status_code)s len: %(body_length)s time: %(wall_seconds).7f {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.887324 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] wsgi_server_debug = False {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 15 10:18:35.888472 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] oslo_concurrency.disable_process_locking = False {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.888590 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] oslo_concurrency.lock_path = /opt/stack/data/neutron/lock {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.888752 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] profiler.connection_string = messaging:// {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.888849 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] profiler.enabled = False {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.888960 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] profiler.es_doc_type = notification {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.889051 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] profiler.es_scroll_size = 10000 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.889166 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] profiler.es_scroll_time = 2m {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.889258 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] profiler.filter_error_trace = False {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.889357 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] profiler.hmac_keys = **** {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.889463 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] profiler.sentinel_service_name = mymaster {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.889569 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] profiler.socket_timeout = 0.1 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.889648 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] profiler.trace_requests = False {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.889761 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] profiler.trace_sqlalchemy = False {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.889876 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] profiler_jaeger.process_tags = {} {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.889954 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] profiler_jaeger.service_name_prefix = None {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.890038 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] profiler_otlp.service_name_prefix = None {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.890147 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.890234 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.890331 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_metrics.metrics_process_name = {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.890443 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.890539 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.890664 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] oslo_policy.enforce_new_defaults = True {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.890744 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] oslo_policy.enforce_scope = True {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.890813 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] oslo_policy.policy_default_rule = default {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.890899 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.891278 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] oslo_policy.policy_file = /etc/neutron/policy.json {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.891278 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.891278 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.891278 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.891278 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.891278 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.891278 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] oslo_policy.remote_timeout = 60.0 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.891518 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.891518 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] service_providers.service_provider = [] {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.891518 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] nova.auth_section = None {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.891518 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] nova.auth_type = password {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.891661 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] nova.auth_url = https://199.204.45.132/identity {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.891661 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] nova.cafile = None {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.891751 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] nova.certfile = None {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.891791 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] nova.collect_timing = False {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.891866 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] nova.default_domain_id = None {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.891910 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] nova.default_domain_name = None {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.891966 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] nova.domain_id = None {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.892025 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] nova.domain_name = None {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.892066 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] nova.endpoint_type = public {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.892169 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] nova.insecure = False {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.892230 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] nova.keyfile = None {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.892289 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] nova.password = **** {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.892353 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] nova.project_domain_id = None {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.892400 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] nova.project_domain_name = Default {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.892440 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] nova.project_id = None {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.892506 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] nova.project_name = service {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.892557 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] nova.region_name = RegionOne {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.892623 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] nova.split_loggers = False {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.892685 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] nova.system_scope = None {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.892824 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] nova.timeout = None {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.892874 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] nova.trust_id = None {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.892934 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] nova.user_domain_id = None {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.892984 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] nova.user_domain_name = Default {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.893041 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] nova.user_id = None {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.893104 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] nova.username = nova {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.893171 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] placement.auth_section = None {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.893239 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] placement.auth_type = password {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.893292 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] placement.cafile = None {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.893348 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] placement.certfile = None {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.893404 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] placement.collect_timing = False {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.893470 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] placement.endpoint_type = public {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.893515 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] placement.insecure = False {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.893558 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] placement.keyfile = None {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.893622 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] placement.region_name = None {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.893695 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] placement.split_loggers = False {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.893741 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] placement.timeout = None {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.893819 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] ironic.auth_section = None {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.893870 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] ironic.auth_type = None {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.893928 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] ironic.cafile = None {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.893984 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] ironic.certfile = None {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.894027 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] ironic.collect_timing = False {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.894101 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] ironic.connect_retries = None {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.894144 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] ironic.connect_retry_delay = None {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.894210 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] ironic.enable_notifications = False {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.894262 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] ironic.endpoint_override = None {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.894309 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] ironic.insecure = False {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.894374 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] ironic.interface = None {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.894420 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] ironic.keyfile = None {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.894481 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] ironic.max_version = None {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.894536 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] ironic.min_version = None {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.894592 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] ironic.region_name = None {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.894650 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] ironic.retriable_status_codes = None {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.894722 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] ironic.service_name = None {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.894771 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] ironic.service_type = None {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.894827 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] ironic.split_loggers = False {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.894890 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] ironic.status_code_retries = None {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.894960 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] ironic.status_code_retry_delay = None {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.895001 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] ironic.timeout = None {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.895060 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] ironic.valid_interfaces = None {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.895126 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] ironic.version = None {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.895186 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] cli_script.dry_run = False {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.895305 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.amqp_auto_delete = False {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.895358 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.amqp_durable_queues = False {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.895395 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.conn_pool_min_size = 2 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.895472 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.conn_pool_ttl = 1200 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.895531 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.direct_mandatory_flag = True {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.895585 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.enable_cancel_on_failover = False {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.895653 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.heartbeat_in_pthread = False {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.895724 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.heartbeat_rate = 3 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.895780 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.heartbeat_timeout_threshold = 60 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.895836 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.hostname = np0000174319 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.895892 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.kombu_compression = None {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.895948 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.kombu_failover_strategy = round-robin {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.896012 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.kombu_missing_consumer_retry_timeout = 60 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.896079 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.kombu_reconnect_delay = 1.0 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.896153 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.kombu_reconnect_splay = 0.0 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.896210 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.processname = neutron-ovn-maintenance-worker {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.896253 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_ha_queues = False {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.896351 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_interval_max = 30 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.896423 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_login_method = AMQPLAIN {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.896423 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_qos_prefetch_count = 0 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.896490 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_quorum_delivery_limit = 0 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.896526 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_quorum_max_memory_bytes = 0 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.896580 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_quorum_max_memory_length = 0 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.896624 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_quorum_queue = False {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.896706 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_retry_backoff = 2 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.896755 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_retry_interval = 1 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.896819 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_stream_fanout = False {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.896878 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_transient_queues_ttl = 1800 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.896945 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_transient_quorum_queue = False {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.896995 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rpc_conn_pool_size = 30 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.897040 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.ssl = False {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.897100 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.ssl_ca_file = {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.897144 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.ssl_cert_file = {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.897207 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.ssl_enforce_fips_mode = False {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.897260 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.ssl_key_file = {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.897317 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.ssl_version = {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.897373 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.use_queue_manager = False {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.897456 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_notifications.driver = [] {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.897502 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_notifications.retry = -1 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.897573 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_notifications.topics = ['notifications'] {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.897639 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_notifications.transport_url = **** {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.897724 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] oslo_reports.file_event_handler = None {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.897790 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] oslo_reports.file_event_handler_interval = 1 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.897847 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] oslo_reports.log_dir = None {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.897919 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] oslo_versionedobjects.fatal_exception_format_errors = False {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.898009 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] QUOTAS.default_quota = -1 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.898076 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] QUOTAS.quota_driver = neutron.db.quota.driver_nolock.DbQuotaNoLockDriver {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.898134 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] QUOTAS.quota_floatingip = 50 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.898200 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] QUOTAS.quota_network = 100 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.898269 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] QUOTAS.quota_port = 500 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.898324 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] QUOTAS.quota_rbac_policy = 10 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.898388 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] QUOTAS.quota_router = 10 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.898451 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] QUOTAS.quota_security_group = 10 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.898519 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] QUOTAS.quota_security_group_rule = 100 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.898570 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] QUOTAS.quota_subnet = 100 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.898638 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] QUOTAS.track_quota_usage = True {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.899023 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] privsep.capabilities = [21, 12, 1, 2, 19] {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.899082 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] privsep.group = None {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.899125 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] privsep.helper_command = None {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.899183 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] privsep.log_daemon_traceback = False {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.899236 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] privsep.logger_name = oslo_privsep.daemon {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.899315 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] privsep.thread_pool_size = 16 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.899363 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] privsep.user = None {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.899433 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] privsep_dhcp_release.capabilities = [21, 12] {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.899478 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] privsep_dhcp_release.group = None {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.899542 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] privsep_dhcp_release.helper_command = None {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.899596 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] privsep_dhcp_release.log_daemon_traceback = False {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.899663 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] privsep_dhcp_release.logger_name = oslo_privsep.daemon {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.899718 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] privsep_dhcp_release.thread_pool_size = 16 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.899774 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] privsep_dhcp_release.user = None {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.899835 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] privsep_ovs_vsctl.capabilities = [21, 12] {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.899889 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] privsep_ovs_vsctl.group = None {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.899943 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] privsep_ovs_vsctl.helper_command = None {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.899995 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] privsep_ovs_vsctl.log_daemon_traceback = False {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.900045 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] privsep_ovs_vsctl.logger_name = oslo_privsep.daemon {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.900100 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] privsep_ovs_vsctl.thread_pool_size = 16 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.900215 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] privsep_ovs_vsctl.user = None {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.900303 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] privsep_namespace.capabilities = [21] {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.900350 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] privsep_namespace.group = None {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.900999 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] privsep_namespace.helper_command = None {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.900999 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] privsep_namespace.log_daemon_traceback = False {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.900999 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] privsep_namespace.logger_name = oslo_privsep.daemon {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.900999 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] privsep_namespace.thread_pool_size = 16 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.900999 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] privsep_namespace.user = None {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.900999 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] privsep_conntrack.capabilities = [12] {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.900999 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] privsep_conntrack.group = None {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.900999 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] privsep_conntrack.helper_command = None {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.900999 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] privsep_conntrack.log_daemon_traceback = False {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.901352 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] privsep_conntrack.logger_name = oslo_privsep.daemon {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.901352 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] privsep_conntrack.thread_pool_size = 16 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.901352 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] privsep_conntrack.user = None {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.901352 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] privsep_link.capabilities = [12, 21] {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.901352 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] privsep_link.group = None {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.901352 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] privsep_link.helper_command = None {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.901352 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] privsep_link.log_daemon_traceback = False {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.901352 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] privsep_link.logger_name = oslo_privsep.daemon {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.901352 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] privsep_link.thread_pool_size = 16 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.901663 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] privsep_link.user = None {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.901663 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] AGENT.root_helper = sudo /opt/stack/data/venv/bin/neutron-rootwrap /etc/neutron/rootwrap.conf {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.901663 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] AGENT.root_helper_daemon = sudo /opt/stack/data/venv/bin/neutron-rootwrap-daemon /etc/neutron/rootwrap.conf {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.901663 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] AGENT.use_helper_for_ns_read = True {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.901663 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] SECURITYGROUP.enable_ipset = True {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.901883 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] SECURITYGROUP.enable_security_group = True {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.901883 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] SECURITYGROUP.firewall_driver = None {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.901883 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] SECURITYGROUP.permitted_ethertypes = [] {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.901986 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] experimental.ipv6_pd_enabled = False {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.901986 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] ml2.extension_drivers = ['port_security', 'qos'] {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.902065 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] ml2.external_network_type = None {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.902106 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] ml2.mechanism_drivers = ['ovn'] {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.902164 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] ml2.overlay_ip_version = 4 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.902220 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] ml2.path_mtu = 0 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.902284 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] ml2.physical_network_mtus = {} {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.902342 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] ml2.project_network_types = ['geneve'] {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.902409 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] ml2.tunnelled_network_rp_name = rp_tunnelled {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.902472 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] ml2.type_drivers = ['local', 'flat', 'vlan', 'geneve'] {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.902567 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] ml2_type_flat.flat_networks = ['public'] {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.902629 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] ml2_type_geneve.max_header_size = 38 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.902716 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] ml2_type_geneve.vni_ranges = ['1:65536'] {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.902788 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] ml2_type_vlan.network_vlan_ranges = ['public'] {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.902873 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] network_log.burst_limit = 25 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.902939 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] network_log.local_output_log_base = None {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.903107 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] network_log.rate_limit = 100 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.903162 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] database.asyncio_connection = **** {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.903224 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] database.asyncio_slave_connection = **** {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.903297 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] database.backend = sqlalchemy {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.903363 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] database.connection = **** {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.903415 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] database.connection_debug = 0 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.903482 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] database.connection_parameters = {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.903550 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] database.connection_recycle_time = 3600 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.903607 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] database.connection_trace = False {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.903664 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] database.db_inc_retry_interval = True {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.903732 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] database.db_max_retries = 20 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.903794 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] database.db_max_retry_interval = 10 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.903855 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] database.db_retry_interval = 1 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.903911 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] database.max_overflow = 50 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.903976 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] database.max_pool_size = 5 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.904033 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] database.max_retries = 10 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.904092 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] database.mysql_sql_mode = TRADITIONAL {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.904217 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] database.mysql_wsrep_sync_wait = None {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.904278 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] database.pool_timeout = None {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.904333 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] database.retry_interval = 10 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.904391 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] database.slave_connection = **** {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.904456 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] database.sqlite_synchronous = True {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.904519 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] database.synchronous_reader = True {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.904576 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] database.use_db_reconnect = False {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.904661 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] ovn.bfd_min_rx = 1000 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.904733 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] ovn.bfd_min_tx = 100 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.904791 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] ovn.bfd_mult = 3 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.904843 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] ovn.broadcast_arps_to_all_routers = True {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.904898 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] ovn.dhcp_default_lease_time = 43200 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.904953 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] ovn.disable_ovn_dhcp_for_baremetal_ports = False {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.905004 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] ovn.dns_records_ovn_owned = False {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.905058 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] ovn.dns_servers = [] {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.905117 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] ovn.enable_distributed_floating_ip = False {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.905163 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] ovn.fdb_age_threshold = 300 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.905222 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] ovn.ha_failover_strategy = normal {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.905273 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] ovn.live_migration_activation_strategy = rarp {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.905367 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] ovn.localnet_learn_fdb = False {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.905476 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] ovn.mac_binding_age_threshold = 300 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.905531 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] ovn.neutron_sync_mode = log {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.905572 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_dhcp4_global_options = {} {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.905621 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_dhcp6_global_options = {} {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.905662 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_l3_scheduler = leastloaded {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.905740 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_metadata_enabled = True {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.905818 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_nb_ca_cert = /opt/stack/data/CA/int-ca/ca-chain.pem {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.905885 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_nb_certificate = /opt/stack/data/CA/int-ca/devstack-cert.crt {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.905935 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_nb_connection = ['ssl:199.204.45.132:6641'] {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.905993 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_nb_private_key = /opt/stack/data/CA/int-ca/private/devstack-cert.key {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.906049 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_router_indirect_snat = False {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.906103 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_sb_ca_cert = /opt/stack/data/CA/int-ca/ca-chain.pem {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.906150 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_sb_certificate = /opt/stack/data/CA/int-ca/devstack-cert.crt {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.906209 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_sb_connection = ['ssl:199.204.45.132:6642'] {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.906263 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_sb_private_key = /opt/stack/data/CA/int-ca/private/devstack-cert.key {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.906318 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovs_create_tap = False {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.906383 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovsdb_connection_timeout = 180 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.906443 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovsdb_log_level = INFO {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.906487 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovsdb_probe_interval = 60000 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.906545 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovsdb_retry_max_interval = 180 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.906599 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] ovn.stateless_nat_enabled = False {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.906663 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] ovn.vhost_sock_dir = /var/run/openvswitch {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.906751 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] OVS.bridge_mac_table_size = 50000 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.906817 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] OVS.igmp_flood = False {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.906874 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] OVS.igmp_flood_reports = True {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.906930 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] OVS.igmp_flood_unregistered = False {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.906986 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] OVS.igmp_snooping_enable = False {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.907046 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] OVS.ovsdb_timeout = 10 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.907103 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] ovn_nb_global.fdb_removal_limit = 0 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.907169 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] ovn_nb_global.ignore_lsp_down = False {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.907230 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] ovn_nb_global.mac_binding_removal_limit = 0 {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 15 10:18:35.907280 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon.oslo_config_glue [-] ******************************************************************************** {{(pid=105458) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3013}} May 15 10:18:35.907488 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG cotyledon._service_worker [-] Run service ServiceWrapper(0) [105458] {{(pid=105458) wait_forever /opt/stack/data/venv/lib/python3.12/site-packages/cotyledon/_service_worker.py:218}} May 15 10:18:35.908946 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron_lib.callbacks.manager [-] Publish callbacks ['neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver.OVNMechanismDriver.post_fork_initialize-178996', 'neutron.services.ovn_l3.plugin.OVNL3RouterPlugin._post_fork_initialize-14341'] for process (None), after_init {{(pid=105458) _notify_loop /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:184}} May 15 10:18:35.910945 np0000174319 neutron-ovn-maintenance-worker[105458]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Getting OvsdbNbOvnIdl for MaintenanceWorker with retry May 15 10:18:36.007200 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl [-] Created lookup_table index Logical_Switch.name {{(pid=105458) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:87}} May 15 10:18:36.007200 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl [-] Created lookup_table index Logical_Router.name {{(pid=105458) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:87}} May 15 10:18:36.007538 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl [-] Created lookup_table index Load_Balancer.name {{(pid=105458) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:87}} May 15 10:18:36.007538 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Copp.name {{(pid=105458) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} May 15 10:18:36.007538 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Meter.name {{(pid=105458) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} May 15 10:18:36.007538 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Load_Balancer_Group.name {{(pid=105458) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} May 15 10:18:36.007538 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index HA_Chassis_Group.name {{(pid=105458) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} May 15 10:18:36.007974 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Address_Set.name {{(pid=105458) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} May 15 10:18:36.007974 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Connection.target {{(pid=105458) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} May 15 10:18:36.007974 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Gateway_Chassis.name {{(pid=105458) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} May 15 10:18:36.007974 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Mirror.name {{(pid=105458) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} May 15 10:18:36.007974 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Chassis_Template_Var.chassis {{(pid=105458) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} May 15 10:18:36.007974 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Logical_Router_Port.name {{(pid=105458) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} May 15 10:18:36.007974 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Port_Group.name {{(pid=105458) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} May 15 10:18:36.008469 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Logical_Switch_Port.name {{(pid=105458) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} May 15 10:18:36.010600 np0000174319 neutron-ovn-maintenance-worker[105458]: INFO ovsdbapp.backend.ovs_idl.vlog [-] ssl:199.204.45.132:6641: connecting... May 15 10:18:36.016646 np0000174319 neutron-ovn-maintenance-worker[105458]: INFO ovsdbapp.backend.ovs_idl.vlog [-] ssl:199.204.45.132:6641: connected May 15 10:18:36.044144 np0000174319 neutron-ovn-maintenance-worker[105458]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Getting OvsdbSbOvnIdl for MaintenanceWorker with retry May 15 10:18:36.079083 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl [-] Created lookup_table index Chassis.name {{(pid=105458) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:87}} May 15 10:18:36.079083 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl [-] Created lookup_table index Port_Binding.logical_port {{(pid=105458) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:87}} May 15 10:18:36.079379 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Chassis_Private.name {{(pid=105458) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} May 15 10:18:36.079379 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Datapath_Binding.tunnel_key {{(pid=105458) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} May 15 10:18:36.081591 np0000174319 neutron-ovn-maintenance-worker[105458]: INFO ovsdbapp.backend.ovs_idl.vlog [-] ssl:199.204.45.132:6642: connecting... May 15 10:18:36.086809 np0000174319 neutron-ovn-maintenance-worker[105458]: INFO ovsdbapp.backend.ovs_idl.vlog [-] ssl:199.204.45.132:6642: connected May 15 10:18:36.121786 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > floatingip precommit_update 55550000, False {{(pid=105458) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:36.122013 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > floatingip precommit_delete 55550000, False {{(pid=105458) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:36.122013 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > floatingip before_create 55550000, False {{(pid=105458) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:36.122013 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > floatingip before_update 55550000, False {{(pid=105458) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:36.122177 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port after_update 55550000, False {{(pid=105458) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:36.122177 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port precommit_delete 55550000, False {{(pid=105458) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:36.122373 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron_lib.callbacks.manager [-] Publish callbacks ['neutron.services.portforwarding.drivers.ovn.driver.OVNPortForwarding.register-14568'] for port_forwarding_plugin (None), after_init {{(pid=105458) _notify_loop /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:184}} May 15 10:18:36.122436 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port_forwarding after_create 55550000, False {{(pid=105458) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:36.122535 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > router_interface after_create 55550000, False {{(pid=105458) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:36.122597 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port_forwarding after_update 55550000, False {{(pid=105458) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:36.122643 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port_forwarding after_delete 55550000, False {{(pid=105458) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:36.122743 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > router_interface after_delete 55550000, False {{(pid=105458) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:36.122840 np0000174319 neutron-ovn-maintenance-worker[105458]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.extensions.qos [-] Starting OVNClientQosExtension May 15 10:18:36.122926 np0000174319 neutron-ovn-maintenance-worker[105458]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.extensions.placement [-] Starting OVNClientPlacementExtension May 15 10:18:36.123313 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > segment before_delete 55550000, False {{(pid=105458) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:36.123375 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > subnet precommit_delete_associations 55550000, False {{(pid=105458) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:36.123695 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > segment_host_mapping after_create 55550000, False {{(pid=105458) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:36.123789 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > segment_host_mapping after_delete 55550000, False {{(pid=105458) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:36.123900 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port after_create 55550000, False {{(pid=105458) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:36.124018 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port after_delete 55550000, False {{(pid=105458) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:36.124139 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port after_update 55550000, False {{(pid=105458) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:36.124192 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > segment after_delete 55550000, False {{(pid=105458) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:36.124272 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > subnet after_create 55550000, False {{(pid=105458) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:36.124332 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > subnet after_delete 55550000, False {{(pid=105458) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:36.124394 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > subnet after_update 55550000, False {{(pid=105458) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:36.181361 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > subnet after_create 55550000, False {{(pid=105458) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:36.181361 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > subnet after_delete 55550000, False {{(pid=105458) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:36.181638 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > subnet before_create 55550000, False {{(pid=105458) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:36.181638 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > subnet before_update 55550000, False {{(pid=105458) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:36.183977 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > security_group after_delete 55550000, False {{(pid=105458) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:36.184138 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port after_delete 55550000, False {{(pid=105458) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:36.184138 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron_lib.callbacks.manager [-] Publish callbacks ['neutron.services.logapi.drivers.ovn.driver.DriverBase._register-1362100'] for logging-plugin (None), after_init {{(pid=105458) _notify_loop /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:184}} May 15 10:18:36.184511 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > security_group_rule after_create 55550000, False {{(pid=105458) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:36.184610 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > security_group_rule after_update 55550000, False {{(pid=105458) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:36.184610 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > security_group_rule after_delete 55550000, False {{(pid=105458) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 15 10:18:45.574961 np0000174319 neutron-ovn-maintenance-worker[104616]: DEBUG dbcounter [-] [104616] Writing DB stats neutron:SELECT=4 {{(pid=104616) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} May 15 10:18:46.186980 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [-] OVN-Northbound DB sync process started @ 2026-05-15 10:18:46.186605 {{(pid=105458) do_sync /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:99}} May 15 10:18:46.189099 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-91bc5273-8551-4913-ae31-04aad06291a7 None None] OVN-NB Sync port groups started @ 2026-05-15 10:18:46.188647 {{(pid=105458) sync_port_groups /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:132}} May 15 10:18:46.210159 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG dbcounter [-] [105458] Writer thread running {{(pid=105458) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:102}} May 15 10:18:46.224803 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-91bc5273-8551-4913-ae31-04aad06291a7 None None] OVN-NB Sync port groups completed @ 2026-05-15 10:18:46.224507 {{(pid=105458) sync_port_groups /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:196}} May 15 10:18:46.224964 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-91bc5273-8551-4913-ae31-04aad06291a7 None None] OVN-NB Sync networks, ports and DHCP options started @ 2026-05-15 10:18:46.224781 {{(pid=105458) sync_networks_ports_and_dhcp_opts /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1152}} May 15 10:18:46.565936 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.services.segments.db [None req-91bc5273-8551-4913-ae31-04aad06291a7 None None] neutron.services.segments.plugin.Plugin method get_segments called with arguments (,) {'filters': {'network_id': ['2bfe92bb-123c-4fb0-89e0-38e15f498891'], 'is_dynamic': False}} {{(pid=105458) wrapper /opt/stack/data/venv/lib/python3.12/site-packages/oslo_log/helpers.py:77}} May 15 10:18:46.572212 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-91bc5273-8551-4913-ae31-04aad06291a7 None None] OVN-NB Sync metadata ports started {{(pid=105458) _sync_metadata_ports /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1097}} May 15 10:18:46.639196 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-91bc5273-8551-4913-ae31-04aad06291a7 None None] OVN-NB Sync metadata ports completed {{(pid=105458) _sync_metadata_ports /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1149}} May 15 10:18:46.639386 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-91bc5273-8551-4913-ae31-04aad06291a7 None None] OVN-NB Sync DHCP options for Neutron subnets started {{(pid=105458) _sync_subnet_dhcp_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:949}} May 15 10:18:46.697178 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-91bc5273-8551-4913-ae31-04aad06291a7 None None] OVN-NB Sync DHCP options for Neutron subnets completed @ 2026-05-15 10:18:46.696764 {{(pid=105458) _sync_subnet_dhcp_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1022}} May 15 10:18:46.697804 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] There are no commands to commit {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:78}} May 15 10:18:46.698119 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-91bc5273-8551-4913-ae31-04aad06291a7 None None] OVN-NB Sync DHCP options for Neutron ports with extra dhcp options assigned started {{(pid=105458) _sync_port_dhcp_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1027}} May 15 10:18:46.698257 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-91bc5273-8551-4913-ae31-04aad06291a7 None None] OVN-NB Sync DHCP options for Neutron ports with extra dhcp options assigned completed {{(pid=105458) _sync_port_dhcp_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1086}} May 15 10:18:46.698427 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-91bc5273-8551-4913-ae31-04aad06291a7 None None] OVN-NB Sync networks, ports and DHCP options completed @ 2026-05-15 10:18:46.698257 {{(pid=105458) sync_networks_ports_and_dhcp_opts /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1321}} May 15 10:18:46.698612 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-91bc5273-8551-4913-ae31-04aad06291a7 None None] OVN-NB Sync ACLs started @ 2026-05-15 10:18:46.698459 {{(pid=105458) sync_acls /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:239}} May 15 10:18:46.754917 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-91bc5273-8551-4913-ae31-04aad06291a7 None None] OVN-NB Sync ACLs completed @ 2026-05-15 10:18:46.754555 {{(pid=105458) sync_acls /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:431}} May 15 10:18:46.755221 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-91bc5273-8551-4913-ae31-04aad06291a7 None None] OVN-NB Sync routers and router ports started @ 2026-05-15 10:18:46.754909 {{(pid=105458) sync_routers_and_rports /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:736}} May 15 10:18:46.824797 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] There are no commands to commit {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:78}} May 15 10:18:46.825184 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-91bc5273-8551-4913-ae31-04aad06291a7 None None] OVN-NB Sync routers and router ports completed @ 2026-05-15 10:18:46.824803 {{(pid=105458) sync_routers_and_rports /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:944}} May 15 10:18:46.825294 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-91bc5273-8551-4913-ae31-04aad06291a7 None None] OVN-NB Sync port QoS policies started @ 2026-05-15 10:18:46.825170 {{(pid=105458) sync_port_qos_policies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1369}} May 15 10:18:46.825427 np0000174319 neutron-ovn-maintenance-worker[105458]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.extensions.qos [None req-91bc5273-8551-4913-ae31-04aad06291a7 None None] Starting OVNClientQosExtension May 15 10:18:46.841655 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-91bc5273-8551-4913-ae31-04aad06291a7 None None] OVN-NB Sync port QoS policies completed @ 2026-05-15 10:18:46.841392 {{(pid=105458) sync_port_qos_policies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1392}} May 15 10:18:46.841841 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-91bc5273-8551-4913-ae31-04aad06291a7 None None] OVN-NB Sync Floating IP QoS policies started @ 2026-05-15 10:18:46.841657 {{(pid=105458) sync_fip_qos_policies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1397}} May 15 10:18:46.841932 np0000174319 neutron-ovn-maintenance-worker[105458]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.extensions.qos [None req-91bc5273-8551-4913-ae31-04aad06291a7 None None] Starting OVNClientQosExtension May 15 10:18:46.871331 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-91bc5273-8551-4913-ae31-04aad06291a7 None None] OVN-NB Sync Floating IP QoS policies completed @ 2026-05-15 10:18:46.871008 {{(pid=105458) sync_fip_qos_policies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1420}} May 15 10:18:46.871531 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-91bc5273-8551-4913-ae31-04aad06291a7 None None] OVN-NB Sync Floating IP NAT rules started @ 2026-05-15 10:18:46.871289 {{(pid=105458) sync_fip_dnat_rules /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1425}} May 15 10:18:46.871682 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-91bc5273-8551-4913-ae31-04aad06291a7 None None] OVN-NB Sync Floating IP NAT rules completed @ 2026-05-15 10:18:46.871562 {{(pid=105458) sync_fip_dnat_rules /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1451}} May 15 10:18:46.871866 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-91bc5273-8551-4913-ae31-04aad06291a7 None None] OVN-Northbound DB sync process completed @ 2026-05-15 10:18:46.871689 {{(pid=105458) do_sync /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:112}} May 15 10:18:56.188292 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [-] OVN-Southbound DB sync process started @ 2026-05-15 10:18:56.187720 {{(pid=105458) do_sync /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1480}} May 15 10:18:56.189250 np0000174319 neutron-ovn-maintenance-worker[105458]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.extensions.qos [-] Starting OVNClientQosExtension May 15 10:18:56.190519 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-3afd3a7b-b3f8-4e42-b002-973c2ebcfcfe None None] OVN-SB Sync hostname and physical networks started @ 2026-05-15 10:18:56.190129 {{(pid=105458) sync_hostname_and_physical_networks /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1511}} May 15 10:18:56.195602 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.add_provnet_ext_id_to_localnet_ports {{(pid=105458) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} May 15 10:18:56.195791 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_baremetal_ports_dhcp_options {{(pid=105458) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} May 15 10:18:56.195932 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_dhcp_options_consistency {{(pid=105458) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} May 15 10:18:56.196065 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_fair_meter_consistency {{(pid=105458) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} May 15 10:18:56.196196 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_fdb_aging_settings {{(pid=105458) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} May 15 10:18:56.196311 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_for_igmp_snoop_support {{(pid=105458) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} May 15 10:18:56.196508 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_for_inconsistencies {{(pid=105458) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} May 15 10:18:56.196728 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_global_dhcp_opts {{(pid=105458) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} May 15 10:18:56.197522 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_localnet_port_has_learn_fdb {{(pid=105458) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} May 15 10:18:56.197723 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_network_broadcast_arps_to_all_routers {{(pid=105458) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} May 15 10:18:56.197920 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_provider_distributed_ports {{(pid=105458) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} May 15 10:18:56.198081 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_redirect_type_router_gateway_ports {{(pid=105458) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} May 15 10:18:56.198281 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip {{(pid=105458) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} May 15 10:18:56.198422 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.cleanup_old_hash_ring_nodes {{(pid=105458) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} May 15 10:18:56.198574 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.configure_nb_global {{(pid=105458) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} May 15 10:18:56.198748 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.migrate_lrp_gateway_chassis_to_ha_chassis_group {{(pid=105458) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} May 15 10:18:56.198920 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.set_fip_distributed_flag {{(pid=105458) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} May 15 10:18:56.199117 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.set_network_type_and_physnet {{(pid=105458) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} May 15 10:18:56.199272 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.set_ovn_owned_dns_option {{(pid=105458) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} May 15 10:18:56.199438 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.update_ha_failover {{(pid=105458) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} May 15 10:18:56.199564 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.update_lrouter_ports_ext_ids_name_prefix {{(pid=105458) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} May 15 10:18:56.199812 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.update_mac_aging_settings {{(pid=105458) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} May 15 10:18:56.200024 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.update_router_distributed_flag {{(pid=105458) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} May 15 10:18:56.200230 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.update_router_static_routes {{(pid=105458) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} May 15 10:18:56.200439 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.update_security_group_with_address_group {{(pid=105458) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} May 15 10:18:56.200649 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.update_virtual_port_parent_hostname {{(pid=105458) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} May 15 10:18:56.203437 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.add_provnet_ext_id_to_localnet_ports' {{(pid=105458) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} May 15 10:18:56.204158 np0000174319 neutron-ovn-maintenance-worker[105458]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver [-] Maintenance task thread has started May 15 10:18:56.205905 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.agent.neutron_agent [None req-3afd3a7b-b3f8-4e42-b002-973c2ebcfcfe None None] Agents populated in the agent cache: {{(pid=105458) populate /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/agent/neutron_agent.py:309}} May 15 10:18:56.207268 np0000174319 neutron-ovn-maintenance-worker[105458]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver [-] MaintenanceWorker process has finished the post initialization May 15 10:18:56.207611 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_baremetal_ports_dhcp_options' {{(pid=105458) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} May 15 10:18:56.209989 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.agent.neutron_agent [None req-3afd3a7b-b3f8-4e42-b002-973c2ebcfcfe None None] - Agent type: OVN Controller Gateway agent, hostname: np0000174319, name: 6f338c9e-6c81-4155-b1f6-9ce61b08c0bc {{(pid=105458) populate /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/agent/neutron_agent.py:311}} May 15 10:18:56.211188 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_dhcp_options_consistency' {{(pid=105458) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} May 15 10:18:56.213728 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.agent.neutron_agent [None req-3afd3a7b-b3f8-4e42-b002-973c2ebcfcfe None None] - Agent type: OVN Neutron agent, hostname: np0000174319, name: 6f338c9e-6c81-4155-b1f6-9ce61b08c0bc {{(pid=105458) populate /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/agent/neutron_agent.py:311}} May 15 10:18:56.219568 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fair_meter_consistency' {{(pid=105458) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} May 15 10:18:56.220337 np0000174319 neutron-ovn-maintenance-worker[105458]: WARNING neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-3afd3a7b-b3f8-4e42-b002-973c2ebcfcfe None None] New host np0000174319 found in OVN SB DB, but not in Neutron. Add its SegmentHostMapping in Neutron May 15 10:18:56.221315 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105458) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} May 15 10:18:56.221960 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.add_provnet_ext_id_to_localnet_ports' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=105458) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} May 15 10:18:56.222092 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_for_igmp_snoop_support' {{(pid=105458) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} May 15 10:18:56.222584 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_for_inconsistencies' {{(pid=105458) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} May 15 10:18:56.222780 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_global_dhcp_opts' {{(pid=105458) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} May 15 10:18:56.222951 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_localnet_port_has_learn_fdb' {{(pid=105458) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} May 15 10:18:56.223099 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_network_broadcast_arps_to_all_routers' {{(pid=105458) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} May 15 10:18:56.223306 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_provider_distributed_ports' {{(pid=105458) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} May 15 10:18:56.223455 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_redirect_type_router_gateway_ports' {{(pid=105458) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} May 15 10:18:56.223626 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105458) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} May 15 10:18:56.224242 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_baremetal_ports_dhcp_options {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:18:56.224366 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Sync baremetal port DHCP options with configuration. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:18:56.224762 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.cleanup_old_hash_ring_nodes' {{(pid=105458) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} May 15 10:18:56.224939 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.configure_nb_global' {{(pid=105458) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} May 15 10:18:56.225100 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.migrate_lrp_gateway_chassis_to_ha_chassis_group' {{(pid=105458) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} May 15 10:18:56.225229 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105458) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} May 15 10:18:56.225373 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_network_type_and_physnet' {{(pid=105458) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} May 15 10:18:56.225511 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_ovn_owned_dns_option' {{(pid=105458) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} May 15 10:18:56.225665 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105458) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} May 15 10:18:56.225889 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_lrouter_ports_ext_ids_name_prefix' {{(pid=105458) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} May 15 10:18:56.226075 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105458) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} May 15 10:18:56.226234 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_router_distributed_flag' {{(pid=105458) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} May 15 10:18:56.226387 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_router_static_routes' {{(pid=105458) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} May 15 10:18:56.226560 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_security_group_with_address_group' {{(pid=105458) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} May 15 10:18:56.226757 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_virtual_port_parent_hostname' {{(pid=105458) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} May 15 10:18:56.237748 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG oslo_concurrency.lockutils [None req-3afd3a7b-b3f8-4e42-b002-973c2ebcfcfe None None] Acquiring lock "update_segment_host_mapping" by "neutron.services.segments.db.update_segment_host_mapping" {{(pid=105458) inner /opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/lockutils.py:506}} May 15 10:18:56.237748 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG oslo_concurrency.lockutils [None req-3afd3a7b-b3f8-4e42-b002-973c2ebcfcfe None None] Lock "update_segment_host_mapping" acquired by "neutron.services.segments.db.update_segment_host_mapping" :: waited 0.000s {{(pid=105458) inner /opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/lockutils.py:519}} May 15 10:18:56.280666 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron_lib.callbacks.manager [None req-3afd3a7b-b3f8-4e42-b002-973c2ebcfcfe None None] Publish callbacks ['neutron.plugins.ml2.plugin.DhcpAgentSchedulerDbMixin.auto_schedule_new_network_segments-2755275', 'neutron.services.segments.plugin.NovaSegmentNotifier._notify_host_addition_to_aggregate-526698'] for segment_host_mapping (None), after_create {{(pid=105458) _notify_loop /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:184}} May 15 10:18:56.315802 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.services.segments.db [None req-3afd3a7b-b3f8-4e42-b002-973c2ebcfcfe None None] Segments {'4efbbdff-5d8b-4960-b5c4-e85c093418fe'} mapped to the host np0000174319 {{(pid=105458) update_segment_host_mapping /opt/stack/neutron/neutron/services/segments/db.py:249}} May 15 10:18:56.322786 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG oslo_concurrency.lockutils [None req-3afd3a7b-b3f8-4e42-b002-973c2ebcfcfe None None] Lock "update_segment_host_mapping" "released" by "neutron.services.segments.db.update_segment_host_mapping" :: held 0.085s {{(pid=105458) inner /opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/lockutils.py:538}} May 15 10:18:56.322990 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-3afd3a7b-b3f8-4e42-b002-973c2ebcfcfe None None] OVN-SB Sync hostname and physical networks completed @ 2026-05-15 10:18:56.322779 {{(pid=105458) sync_hostname_and_physical_networks /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1538}} May 15 10:18:56.324432 np0000174319 neutron-ovn-maintenance-worker[105458]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_baremetal_ports_dhcp_options finished in 0.099 seconds May 15 10:18:56.325547 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_baremetal_ports_dhcp_options' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=105458) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} May 15 10:18:56.325925 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_dhcp_options_consistency {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:18:56.331406 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Backfill network_id on DHCP_Options external_ids. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:18:56.389901 np0000174319 neutron-ovn-maintenance-worker[105458]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_dhcp_options_consistency finished in 0.056 seconds May 15 10:18:56.390406 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_dhcp_options_consistency' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=105458) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} May 15 10:18:56.390594 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_fair_meter_consistency {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:18:56.390774 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Refresh ACL logging fair meter after configuration reload. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:18:56.391134 np0000174319 neutron-ovn-maintenance-worker[105458]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_fair_meter_consistency finished in 0.000 seconds May 15 10:18:56.391284 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fair_meter_consistency' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=105458) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} May 15 10:18:56.391443 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:18:56.391550 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:18:56.394618 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:18:56.510314 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:18:56.510907 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:18:56.514659 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:18:56.514659 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:18:56.514659 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:18:56.514659 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:18:56.514659 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:18:56.514659 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:18:56.514659 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:18:56.514659 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:18:56.522779 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_fdb_aging_settings failed after 0.123 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:18:56.522779 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:18:56.522779 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:18:56.522779 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:18:56.522779 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:18:56.522779 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:18:56.522779 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:18:56.522779 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:18:56.522779 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:18:56.522779 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:18:56.522779 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:18:56.522779 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:18:56.523172 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:18:56.523172 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:18:56.523172 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:18:56.523172 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:18:56.523172 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:18:56.523172 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:18:56.523172 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:18:56.523172 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:18:56.523172 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:18:56.523172 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:18:56.523172 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:18:56.523172 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:18:56.523172 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:18:56.523534 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:18:56.523534 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:18:56.529043 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call immediate 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:18:56.529043 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:18:56.529043 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:18:56.529043 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:18:56.529043 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:18:56.529043 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:18:56.529043 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:18:56.529043 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:18:56.529043 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:18:56.529043 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:18:56.529043 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:18:56.529043 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:18:56.529043 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:18:56.529043 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:18:56.529043 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:18:56.529043 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:18:56.529866 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:18:56.529866 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:18:56.529866 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:18:56.529866 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:18:56.529866 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:18:56.529866 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:18:56.529866 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:18:56.529866 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:18:56.529866 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:18:56.529866 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:18:56.529866 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:18:56.529866 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:18:56.529866 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:18:56.529866 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:18:56.529866 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:18:56.529866 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:18:56.529866 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:18:56.529866 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:18:56.529866 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:18:56.530972 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:18:56.530972 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:18:56.530972 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:18:56.530972 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_for_igmp_snoop_support {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:18:56.530972 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Check for IGMP snoop support. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:18:56.530972 np0000174319 neutron-ovn-maintenance-worker[105458]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_for_igmp_snoop_support finished in 0.000 seconds May 15 10:18:56.530972 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_for_igmp_snoop_support' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=105458) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} May 15 10:18:56.530972 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_for_inconsistencies {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:18:56.531485 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Checking Neutron and OVN revision consistency. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:18:56.570125 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Maintenance task: No inconsistencies found. Skipping {{(pid=105458) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:437}} May 15 10:18:56.570410 np0000174319 neutron-ovn-maintenance-worker[105458]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_for_inconsistencies finished in 0.040 seconds May 15 10:18:56.570800 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_global_dhcp_opts {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:18:56.570993 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Check global DHCP options consistency. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:18:56.571431 np0000174319 neutron-ovn-maintenance-worker[105458]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_global_dhcp_opts finished in 0.000 seconds May 15 10:18:56.572213 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_global_dhcp_opts' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=105458) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} May 15 10:18:56.572539 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_localnet_port_has_learn_fdb {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:18:56.572821 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Ensure localnet ports have learn-fdb set per configuration. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:18:56.587467 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] There are no commands to commit {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:78}} May 15 10:18:56.588296 np0000174319 neutron-ovn-maintenance-worker[105458]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_localnet_port_has_learn_fdb finished in 0.015 seconds May 15 10:18:56.590346 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_localnet_port_has_learn_fdb' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=105458) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} May 15 10:18:56.590642 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_network_broadcast_arps_to_all_routers {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:18:56.590851 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Sync broadcast-arps-to-all-routers on external networks. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:18:56.715263 np0000174319 neutron-ovn-maintenance-worker[105458]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_network_broadcast_arps_to_all_routers finished in 0.124 seconds May 15 10:18:56.717328 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_network_broadcast_arps_to_all_routers' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=105458) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} May 15 10:18:56.717620 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_provider_distributed_ports {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:18:56.717835 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Align reside-on-redirect-chassis on VLAN/FLAT distributed ports. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:18:56.833557 np0000174319 neutron-ovn-maintenance-worker[105458]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_provider_distributed_ports finished in 0.115 seconds May 15 10:18:56.834508 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_provider_distributed_ports' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=105458) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} May 15 10:18:56.834596 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_redirect_type_router_gateway_ports {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:18:56.834710 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Align router gateway redirect-type for provider VLAN/FLAT networks. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:18:56.894981 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] There are no commands to commit {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:78}} May 15 10:18:56.895170 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-3afd3a7b-b3f8-4e42-b002-973c2ebcfcfe None None] OVN-Southbound DB sync process completed @ 2026-05-15 10:18:56.895005 {{(pid=105458) do_sync /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1488}} May 15 10:18:56.937951 np0000174319 neutron-ovn-maintenance-worker[105458]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_redirect_type_router_gateway_ports finished in 0.103 seconds May 15 10:18:56.937951 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_redirect_type_router_gateway_ports' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=105458) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} May 15 10:18:56.938198 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:18:56.938198 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:18:56.939616 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b027d63e-13a5-445b-84a0-8ae6223e05d5, routes=[], if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:18:56.940072 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:18:56.941162 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:18:56.941162 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:18:56.941162 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:18:56.941162 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:18:56.941162 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:18:56.941162 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:18:56.941162 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:18:56.941162 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:18:56.941991 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:18:56.941991 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:18:56.941991 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:18:56.941991 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:18:56.941991 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:18:56.941991 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:18:56.941991 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:18:56.941991 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:18:56.941991 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:18:56.941991 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:18:56.941991 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:18:56.941991 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:18:56.942766 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:18:56.942766 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:18:56.942766 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:18:56.942766 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:18:56.942766 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:18:56.942766 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:18:56.942766 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:18:56.942766 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:18:56.942766 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:18:56.942766 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:18:56.942766 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:18:56.942766 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:18:56.942766 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:18:56.943698 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:18:56.943698 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:18:56.943698 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call immediate 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:18:56.943698 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:18:56.943698 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:18:56.943698 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:18:56.943698 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:18:56.943698 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:18:56.943698 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:18:56.943698 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:18:56.943698 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:18:56.943698 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:18:56.943698 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:18:56.943698 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:18:56.943698 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:18:56.944477 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:18:56.944477 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:18:56.944477 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:18:56.944477 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:18:56.944477 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:18:56.944477 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:18:56.944477 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:18:56.944477 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:18:56.944477 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:18:56.944477 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:18:56.944477 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:18:56.944477 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:18:56.944477 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:18:56.944477 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:18:56.944477 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:18:56.944477 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:18:56.944477 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:18:56.944477 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:18:56.944477 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:18:56.945154 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:18:56.945154 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:18:56.945154 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:18:56.945154 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:18:56.945154 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:18:56.945154 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:18:56.945154 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: cleanup_old_hash_ring_nodes {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:18:56.945154 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Purge stale hash ring nodes older than five days. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:18:56.947346 np0000174319 neutron-ovn-maintenance-worker[105458]: INFO neutron.db.ovn_hash_ring_db [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Cleaned up Hash Ring nodes older than 5 days May 15 10:18:56.947478 np0000174319 neutron-ovn-maintenance-worker[105458]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task cleanup_old_hash_ring_nodes finished in 0.004 seconds May 15 10:18:56.947695 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: configure_nb_global {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:18:56.947817 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply ovn_nb_global settings to NB_Global options. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:18:56.948075 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'ignore_lsp_down': 'false', 'fdb_removal_limit': '0', 'mac_binding_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:18:56.948841 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'ignore_lsp_down': 'false', 'fdb_removal_limit': '0', 'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:18:56.949335 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:18:56.950139 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:18:56.950139 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:18:56.950139 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:18:56.950139 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:18:56.950139 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:18:56.950139 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:18:56.950139 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:18:56.950139 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:18:56.951082 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:18:56.951082 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:18:56.951082 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:18:56.951082 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:18:56.951082 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:18:56.951082 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:18:56.951082 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:18:56.951082 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:18:56.951082 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:18:56.951082 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:18:56.951082 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:18:56.951082 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:18:56.951082 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:18:56.951082 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:18:56.951082 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:18:56.951610 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:18:56.951610 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:18:56.951610 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:18:56.951610 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:18:56.951610 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:18:56.951610 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:18:56.951610 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:18:56.951610 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:18:56.951610 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:18:56.952130 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task configure_nb_global failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:18:56.952130 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:18:56.952130 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:18:56.952130 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:18:56.952130 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:18:56.952130 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1038, in configure_nb_global May 15 10:18:56.952130 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 15 10:18:56.952130 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:18:56.952130 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:18:56.952130 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:18:56.952130 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:18:56.952579 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:18:56.952579 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:18:56.952579 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:18:56.952579 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:18:56.952579 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:18:56.952579 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:18:56.952579 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:18:56.952579 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:18:56.952579 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:18:56.952579 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:18:56.952579 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:18:56.952579 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:18:56.952579 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:18:56.952579 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:18:56.953018 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:18:56.953018 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:18:56.953018 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:18:56.953018 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:18:56.953260 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call immediate 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.configure_nb_global' (it runs every 86400.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:18:56.953260 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:18:56.953260 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:18:56.953260 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:18:56.953260 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:18:56.953260 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:18:56.953260 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:18:56.953260 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:18:56.953260 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:18:56.953260 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:18:56.953260 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:18:56.953260 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:18:56.953260 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:18:56.953260 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:18:56.953260 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:18:56.953260 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:18:56.953850 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1038, in configure_nb_global May 15 10:18:56.953850 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 15 10:18:56.953850 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:18:56.953850 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:18:56.953850 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:18:56.953850 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:18:56.953850 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:18:56.953850 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:18:56.953850 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:18:56.953850 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:18:56.953850 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:18:56.953850 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:18:56.953850 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:18:56.953850 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:18:56.953850 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:18:56.953850 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:18:56.953850 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:18:56.953850 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:18:56.953850 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:18:56.954485 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:18:56.954485 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:18:56.954485 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:18:56.954485 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:18:56.954485 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:18:56.954485 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] There are no commands to commit {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:78}} May 15 10:18:56.954485 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.migrate_lrp_gateway_chassis_to_ha_chassis_group' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=105458) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} May 15 10:18:56.954485 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:18:56.954485 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:18:56.955067 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105458) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 15 10:18:56.955173 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:18:56.955391 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:18:56.956039 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:18:56.956039 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:18:56.956039 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:18:56.956039 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:18:56.956039 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:18:56.956039 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:18:56.956039 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:18:56.956039 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:18:56.956853 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:18:56.956853 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:18:56.956853 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:18:56.956853 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:18:56.956853 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:18:56.956853 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:18:56.956853 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:18:56.956853 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:18:56.956853 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:18:56.956853 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:18:56.956853 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:18:56.956853 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:18:56.956853 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:18:56.956853 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:18:56.956853 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:18:56.957316 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:18:56.957316 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:18:56.957316 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:18:56.957316 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:18:56.957316 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:18:56.957316 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:18:56.957316 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:18:56.957316 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:18:56.957316 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:18:56.957661 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:18:56.957661 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:18:56.957661 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:18:56.957661 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:18:56.957661 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:18:56.957661 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:18:56.957661 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:18:56.957661 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 15 10:18:56.957661 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:18:56.957661 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:18:56.957661 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:18:56.957661 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:18:56.958039 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:18:56.958039 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:18:56.958039 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:18:56.958039 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:18:56.958039 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:18:56.958039 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:18:56.958039 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:18:56.958039 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:18:56.958039 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:18:56.958039 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:18:56.958039 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:18:56.958039 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:18:56.958039 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:18:56.958039 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:18:56.958421 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:18:56.958421 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:18:56.958421 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:18:56.958421 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:18:56.958670 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call immediate 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:18:56.958670 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:18:56.958670 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:18:56.958670 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:18:56.958670 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:18:56.958670 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:18:56.958670 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:18:56.958670 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:18:56.958670 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:18:56.958670 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:18:56.958670 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:18:56.958670 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:18:56.958670 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:18:56.958670 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:18:56.958670 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:18:56.958670 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:18:56.959117 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:18:56.959117 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:18:56.959117 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^ May 15 10:18:56.959117 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:18:56.959117 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:18:56.959117 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:18:56.959117 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:18:56.959117 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:18:56.959117 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:18:56.959117 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:18:56.959117 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:18:56.959117 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:18:56.959117 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:18:56.959117 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:18:56.959117 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:18:56.959117 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:18:56.959117 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:18:56.959117 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:18:56.959622 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:18:56.959622 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:18:56.959622 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:18:56.959622 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:18:56.959622 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:18:56.959622 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:18:56.959622 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:18:56.959622 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: set_network_type_and_physnet {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:18:56.959622 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Set network type and physnet external_ids on logical switches. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:18:56.965671 np0000174319 neutron-ovn-maintenance-worker[105458]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task set_network_type_and_physnet finished in 0.006 seconds May 15 10:18:56.965832 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_network_type_and_physnet' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=105458) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} May 15 10:18:56.966030 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: set_ovn_owned_dns_option {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:18:56.966149 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply ovn_owned option on DNS records per configuration. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:18:56.966394 np0000174319 neutron-ovn-maintenance-worker[105458]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task set_ovn_owned_dns_option finished in 0.000 seconds May 15 10:18:56.966527 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_ovn_owned_dns_option' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=105458) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} May 15 10:18:56.966790 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_ha_failover {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:18:56.966790 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:18:56.967007 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:18:56.967753 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:18:56.968076 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:18:56.968885 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:18:56.968885 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:18:56.968885 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:18:56.968885 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:18:56.968885 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:18:56.968885 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:18:56.968885 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:18:56.968885 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:18:56.969628 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:18:56.969628 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:18:56.969628 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:18:56.969628 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:18:56.969628 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:18:56.969628 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:18:56.969628 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:18:56.969628 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:18:56.969628 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:18:56.969628 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:18:56.969628 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:18:56.969628 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:18:56.969628 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:18:56.969628 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:18:56.969628 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:18:56.970448 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:18:56.970448 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:18:56.970448 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:18:56.970448 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:18:56.970448 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:18:56.970448 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:18:56.970448 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:18:56.970448 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:18:56.970448 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:18:56.971850 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:18:56.971850 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:18:56.971850 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:18:56.971850 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:18:56.971850 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:18:56.971850 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:18:56.971850 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 15 10:18:56.971850 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:18:56.971850 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:18:56.971850 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:18:56.971850 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:18:56.973108 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:18:56.973108 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:18:56.973108 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:18:56.973108 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:18:56.973108 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:18:56.973108 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:18:56.973108 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:18:56.973108 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:18:56.973108 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:18:56.973108 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:18:56.973108 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:18:56.973108 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:18:56.973108 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:18:56.973108 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:18:56.973749 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:18:56.973749 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:18:56.973749 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:18:56.973749 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:18:56.973749 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call immediate 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:18:56.973749 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:18:56.973749 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:18:56.973749 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:18:56.973749 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:18:56.973749 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:18:56.973749 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:18:56.973749 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:18:56.973749 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:18:56.973749 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:18:56.974413 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:18:56.974413 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:18:56.974413 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:18:56.974413 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:18:56.974413 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:18:56.974413 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:18:56.974413 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:18:56.974413 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 15 10:18:56.974413 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:18:56.974413 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:18:56.974413 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:18:56.974413 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:18:56.974413 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:18:56.974413 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:18:56.974413 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:18:56.974413 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:18:56.974413 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:18:56.974413 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:18:56.974413 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:18:56.975484 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:18:56.975484 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:18:56.975484 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:18:56.975484 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:18:56.975484 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:18:56.975484 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:18:56.975484 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:18:56.975484 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:18:56.975484 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:18:56.975484 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:18:56.975484 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:18:56.975484 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_lrouter_ports_ext_ids_name_prefix {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:18:56.975484 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: ('Check missing prefix router_name', 'in external_ids of LRPs.') {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:18:56.976358 np0000174319 neutron-ovn-maintenance-worker[105458]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_lrouter_ports_ext_ids_name_prefix finished in 0.001 seconds May 15 10:18:56.976358 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_lrouter_ports_ext_ids_name_prefix' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=105458) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} May 15 10:18:56.976358 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:18:56.976358 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:18:56.976358 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:18:56.976886 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:18:56.976886 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:18:56.976886 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:18:56.976886 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:18:56.976886 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:18:56.976886 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:18:56.976886 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:18:56.976886 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:18:56.976886 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:18:56.976886 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:18:56.976886 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:18:56.976886 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:18:56.977628 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:18:56.977628 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:18:56.977628 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:18:56.977628 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:18:56.977628 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:18:56.977628 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:18:56.977628 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:18:56.977628 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:18:56.977628 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:18:56.977628 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:18:56.977628 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:18:56.977628 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:18:56.977628 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:18:56.977628 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:18:56.978023 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:18:56.978023 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:18:56.978023 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:18:56.978023 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:18:56.978023 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:18:56.978023 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:18:56.978023 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:18:56.978023 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:18:56.978023 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:18:56.978023 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:18:56.978023 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:18:56.978023 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:18:56.978370 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call immediate 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:18:56.978370 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:18:56.978370 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:18:56.978370 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:18:56.978370 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:18:56.978370 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:18:56.978370 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:18:56.978370 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:18:56.978370 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:18:56.978370 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:18:56.978370 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:18:56.978370 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:18:56.978370 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:18:56.978370 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:18:56.978370 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:18:56.978370 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:18:56.978819 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:18:56.978819 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:18:56.978819 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:18:56.978819 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:18:56.978819 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:18:56.978819 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:18:56.978819 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:18:56.978819 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:18:56.978819 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:18:56.978819 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:18:56.978819 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:18:56.978819 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:18:56.978819 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:18:56.978819 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:18:56.978819 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:18:56.978819 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:18:56.978819 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:18:56.978819 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:18:56.978819 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:18:56.979347 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:18:56.979347 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:18:56.979347 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:18:56.979347 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_router_distributed_flag {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:18:56.979347 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Sync router.distributed with distributed floating IP config. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:18:56.981953 np0000174319 neutron-ovn-maintenance-worker[105458]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_router_distributed_flag finished in 0.004 seconds May 15 10:18:56.982088 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_router_distributed_flag' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=105458) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} May 15 10:18:56.982236 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_router_static_routes {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:18:56.982343 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Mark Neutron-owned static routes in OVN external_ids. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:18:56.983183 np0000174319 neutron-ovn-maintenance-worker[105458]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_router_static_routes finished in 0.001 seconds May 15 10:18:56.983316 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_router_static_routes' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=105458) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} May 15 10:18:56.983459 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_security_group_with_address_group {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:18:56.983563 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Repair Address_Sets and ACLs for address group rules. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:18:56.993240 np0000174319 neutron-ovn-maintenance-worker[105458]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_security_group_with_address_group finished in 0.010 seconds May 15 10:18:56.993337 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_security_group_with_address_group' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=105458) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} May 15 10:18:56.994160 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_virtual_port_parent_hostname' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=105458) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} May 15 10:18:57.152199 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.event [-] Matched CREATE: LogicalRouterPortEvent(events=('create', 'delete'), table='Logical_Router_Port', conditions=None, old_conditions=None), priority=20 to row=Logical_Router_Port(uuid=9ca04eed-45b8-4004-aece-5c0932c3a7ba, enabled=[], external_ids={'neutron:is_ext_gw': 'False', 'neutron:network_name': 'neutron-2bfe92bb-123c-4fb0-89e0-38e15f498891', 'neutron:revision_number': '1', 'neutron:router_name': 'neutron-b027d63e-13a5-445b-84a0-8ae6223e05d5', 'neutron:subnet_ids': '143d14dd-5fcb-467e-b0e9-e7d959ff62e9'}, gateway_chassis=[], ha_chassis_group=[], ipv6_prefix=[], ipv6_ra_configs={'address_mode': 'slaac', 'mtu': '1372', 'send_periodic': 'true'}, mac='fa:16:3e:0e:3b:34', name='lrp-848aa7f9-490a-4b77-99a1-f44f4ef8054c', networks=['fd66:e1f9:d244::1/64'], options={}, peer=[], status={}) old= {{(pid=105458) matches /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/event.py:55}} May 15 10:18:57.153305 np0000174319 neutron-ovn-maintenance-worker[105458]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.extensions.qos [-] Starting OVNClientQosExtension May 15 10:18:57.198302 np0000174319 neutron-ovn-maintenance-worker[105458]: INFO neutron.common.ovn.utils [-] HA Chassis Group neutron-2bfe92bb-123c-4fb0-89e0-38e15f498891 synchronized; highest priority chassis 6f338c9e-6c81-4155-b1f6-9ce61b08c0bc May 15 10:18:57.199045 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=HA_Chassis_Group, record=neutron-2bfe92bb-123c-4fb0-89e0-38e15f498891, col_values=(('external_ids', {'neutron:availability_zone_hints': '', 'neutron:network_id': '2bfe92bb-123c-4fb0-89e0-38e15f498891', 'neutron:router_id': 'b027d63e-13a5-445b-84a0-8ae6223e05d5'}),), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:18:57.199045 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): HAChassisGroupAddChassisCommand(_result=None, hcg=neutron-2bfe92bb-123c-4fb0-89e0-38e15f498891, chassis=6f338c9e-6c81-4155-b1f6-9ce61b08c0bc, priority=1, columns={}) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:18:57.199045 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:18:57.199627 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): May 15 10:18:57.199627 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:18:57.199627 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:18:57.199627 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:18:57.199627 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:18:57.199627 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:18:57.199627 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:18:57.199627 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:18:57.201089 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.event [-] Unexpected exception in notify_loop: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:18:57.201089 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.event Traceback (most recent call last): May 15 10:18:57.201089 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.event File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/event.py", line 185, in notify_loop May 15 10:18:57.201089 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.event match.run(event, row, updates) May 15 10:18:57.201089 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.event File "/opt/stack/neutron/neutron/services/ovn_l3/ovsdb_monitor.py", line 86, in run May 15 10:18:57.201089 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.event self.l3_plugin._ovn_client.link_network_ha_chassis_group( May 15 10:18:57.201089 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.event File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2416, in link_network_ha_chassis_group May 15 10:18:57.201089 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.event with self._nb_idl.transaction(check_error=True) as txn: May 15 10:18:57.201089 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.event File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:18:57.201089 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.event next(self.gen) May 15 10:18:57.201089 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.event File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:18:57.201089 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.event with super().transaction(*args, **kwargs) as t: May 15 10:18:57.201089 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.event File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:18:57.201089 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.event next(self.gen) May 15 10:18:57.201089 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.event File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:18:57.201089 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.event with self.create_transaction( May 15 10:18:57.201089 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.event File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:18:57.201089 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.event self.result = self.commit() May 15 10:18:57.201089 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.event ^^^^^^^^^^^^^ May 15 10:18:57.201953 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.event File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:18:57.201953 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.event raise result.ex May 15 10:18:57.201953 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.event File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:18:57.201953 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.event txn.results.put(txn.do_commit()) May 15 10:18:57.201953 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.event ^^^^^^^^^^^^^^^ May 15 10:18:57.201953 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.event File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:18:57.201953 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.event raise RuntimeError(msg) May 15 10:18:57.201953 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.event RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:18:57.201953 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.event May 15 10:18:59.291934 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.event [-] Matched DELETE: LogicalRouterPortEvent(events=('create', 'delete'), table='Logical_Router_Port', conditions=None, old_conditions=None), priority=20 to row=Logical_Router_Port(uuid=0b378a2d-35f9-4f78-a745-f118ef740fc0, enabled=[], external_ids={'neutron:is_ext_gw': 'True', 'neutron:network_name': 'neutron-688b09c5-29c4-46f5-b6ee-c145e26e6ef8', 'neutron:revision_number': '1', 'neutron:router_name': 'neutron-b027d63e-13a5-445b-84a0-8ae6223e05d5', 'neutron:subnet_ids': '3174159a-e5b8-4301-a3b4-276f30e0a2aa'}, gateway_chassis=[], ha_chassis_group=bd76db72-8ee8-4ae3-9cf5-55d86d80811b, ipv6_prefix=[], ipv6_ra_configs={}, mac='fa:16:3e:3b:81:65', name='lrp-d3cceca8-3712-454d-81ef-b3e3218fb313', networks=['172.24.5.19/24'], options={'gateway_mtu': '1372', 'reside-on-redirect-chassis': 'true'}, peer=[], status={'hosting-chassis': '6f338c9e-6c81-4155-b1f6-9ce61b08c0bc'}) old= {{(pid=105458) matches /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/event.py:55}} May 15 10:18:59.292804 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.event [-] Matched CREATE: LogicalRouterPortEvent(events=('create', 'delete'), table='Logical_Router_Port', conditions=None, old_conditions=None), priority=20 to row=Logical_Router_Port(uuid=0ea39668-3999-486c-b9fd-4335a77771dd, enabled=[], external_ids={'neutron:is_ext_gw': 'True', 'neutron:network_name': 'neutron-688b09c5-29c4-46f5-b6ee-c145e26e6ef8', 'neutron:revision_number': '8', 'neutron:router_name': 'neutron-b027d63e-13a5-445b-84a0-8ae6223e05d5', 'neutron:subnet_ids': '3174159a-e5b8-4301-a3b4-276f30e0a2aa 085d2899-46e2-4891-a571-bd41d7325127'}, gateway_chassis=[], ha_chassis_group=bd76db72-8ee8-4ae3-9cf5-55d86d80811b, ipv6_prefix=[], ipv6_ra_configs={}, mac='fa:16:3e:3b:81:65', name='lrp-d3cceca8-3712-454d-81ef-b3e3218fb313', networks=['172.24.5.19/24', '2001:db8::1/64'], options={'gateway_mtu': '1372', 'reside-on-redirect-chassis': 'true'}, peer=[], status={}) old= {{(pid=105458) matches /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/event.py:55}} May 15 10:18:59.330196 np0000174319 neutron-ovn-maintenance-worker[105458]: INFO neutron.common.ovn.utils [-] HA Chassis Group neutron-2bfe92bb-123c-4fb0-89e0-38e15f498891 synchronized; highest priority chassis 6f338c9e-6c81-4155-b1f6-9ce61b08c0bc May 15 10:18:59.330979 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=HA_Chassis_Group, record=neutron-2bfe92bb-123c-4fb0-89e0-38e15f498891, col_values=(('external_ids', {'neutron:availability_zone_hints': '', 'neutron:network_id': '2bfe92bb-123c-4fb0-89e0-38e15f498891', 'neutron:router_id': 'b027d63e-13a5-445b-84a0-8ae6223e05d5'}),), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:18:59.331298 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): HAChassisGroupAddChassisCommand(_result=None, hcg=neutron-2bfe92bb-123c-4fb0-89e0-38e15f498891, chassis=6f338c9e-6c81-4155-b1f6-9ce61b08c0bc, priority=1, columns={}) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:18:59.331944 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:18:59.332334 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): May 15 10:18:59.332334 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:18:59.332334 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:18:59.332334 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:18:59.332334 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:18:59.332334 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:18:59.332334 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:18:59.332334 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:18:59.333265 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.event [-] Unexpected exception in notify_loop: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:18:59.333265 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.event Traceback (most recent call last): May 15 10:18:59.333265 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.event File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/event.py", line 185, in notify_loop May 15 10:18:59.333265 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.event match.run(event, row, updates) May 15 10:18:59.333265 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.event File "/opt/stack/neutron/neutron/services/ovn_l3/ovsdb_monitor.py", line 81, in run May 15 10:18:59.333265 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.event self.l3_plugin._ovn_client.update_router_ha_chassis_group( May 15 10:18:59.333265 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.event File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1665, in update_router_ha_chassis_group May 15 10:18:59.333265 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.event self.link_network_ha_chassis_group(context, network_id, router_id) May 15 10:18:59.333265 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.event File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2416, in link_network_ha_chassis_group May 15 10:18:59.333265 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.event with self._nb_idl.transaction(check_error=True) as txn: May 15 10:18:59.333265 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.event File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:18:59.333265 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.event next(self.gen) May 15 10:18:59.333265 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.event File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:18:59.333265 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.event with super().transaction(*args, **kwargs) as t: May 15 10:18:59.333265 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.event File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:18:59.333265 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.event next(self.gen) May 15 10:18:59.333265 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.event File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:18:59.333265 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.event with self.create_transaction( May 15 10:18:59.333994 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.event File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:18:59.333994 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.event self.result = self.commit() May 15 10:18:59.333994 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.event ^^^^^^^^^^^^^ May 15 10:18:59.333994 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.event File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:18:59.333994 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.event raise result.ex May 15 10:18:59.333994 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.event File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:18:59.333994 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.event txn.results.put(txn.do_commit()) May 15 10:18:59.333994 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.event ^^^^^^^^^^^^^^^ May 15 10:18:59.333994 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.event File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:18:59.333994 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.event raise RuntimeError(msg) May 15 10:18:59.333994 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.event RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:18:59.333994 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.event May 15 10:18:59.365028 np0000174319 neutron-ovn-maintenance-worker[105458]: INFO neutron.common.ovn.utils [-] HA Chassis Group neutron-2bfe92bb-123c-4fb0-89e0-38e15f498891 synchronized; highest priority chassis 6f338c9e-6c81-4155-b1f6-9ce61b08c0bc May 15 10:18:59.365967 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=HA_Chassis_Group, record=neutron-2bfe92bb-123c-4fb0-89e0-38e15f498891, col_values=(('external_ids', {'neutron:availability_zone_hints': '', 'neutron:network_id': '2bfe92bb-123c-4fb0-89e0-38e15f498891', 'neutron:router_id': 'b027d63e-13a5-445b-84a0-8ae6223e05d5'}),), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:18:59.367424 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): HAChassisGroupAddChassisCommand(_result=None, hcg=neutron-2bfe92bb-123c-4fb0-89e0-38e15f498891, chassis=6f338c9e-6c81-4155-b1f6-9ce61b08c0bc, priority=1, columns={}) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:18:59.367424 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:18:59.367545 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): May 15 10:18:59.367545 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:18:59.367545 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:18:59.367545 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:18:59.367545 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:18:59.367545 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:18:59.367545 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:18:59.367545 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:18:59.368851 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.event [-] Unexpected exception in notify_loop: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:18:59.368851 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.event Traceback (most recent call last): May 15 10:18:59.368851 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.event File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/event.py", line 185, in notify_loop May 15 10:18:59.368851 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.event match.run(event, row, updates) May 15 10:18:59.368851 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.event File "/opt/stack/neutron/neutron/services/ovn_l3/ovsdb_monitor.py", line 89, in run May 15 10:18:59.368851 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.event self.l3_plugin._ovn_client.update_router_ha_chassis_group( May 15 10:18:59.368851 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.event File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1665, in update_router_ha_chassis_group May 15 10:18:59.368851 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.event self.link_network_ha_chassis_group(context, network_id, router_id) May 15 10:18:59.368851 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.event File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2416, in link_network_ha_chassis_group May 15 10:18:59.368851 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.event with self._nb_idl.transaction(check_error=True) as txn: May 15 10:18:59.368851 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.event File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:18:59.368851 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.event next(self.gen) May 15 10:18:59.368851 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.event File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:18:59.368851 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.event with super().transaction(*args, **kwargs) as t: May 15 10:18:59.368851 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.event File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:18:59.368851 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.event next(self.gen) May 15 10:18:59.368851 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.event File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:18:59.368851 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.event with self.create_transaction( May 15 10:18:59.369427 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.event File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:18:59.369427 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.event self.result = self.commit() May 15 10:18:59.369427 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.event ^^^^^^^^^^^^^ May 15 10:18:59.369427 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.event File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:18:59.369427 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.event raise result.ex May 15 10:18:59.369427 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.event File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:18:59.369427 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.event txn.results.put(txn.do_commit()) May 15 10:18:59.369427 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.event ^^^^^^^^^^^^^^^ May 15 10:18:59.369427 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.event File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:18:59.369427 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.event raise RuntimeError(msg) May 15 10:18:59.369427 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.event RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:18:59.369427 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.event May 15 10:19:01.391862 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:19:01.392612 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:19:01.393245 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:19:01.394285 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:19:01.631865 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:19:01.632193 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:01.633079 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:19:01.633079 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:01.633079 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:19:01.633079 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:19:01.633079 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:01.633079 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:19:01.633079 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:01.633079 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:19:01.634172 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_fdb_aging_settings failed after 0.240 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:01.634172 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:19:01.634172 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:01.634172 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:19:01.634172 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:19:01.634172 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:19:01.634172 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:19:01.634172 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:01.634172 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:01.634172 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:01.634172 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:19:01.634172 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:01.635003 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:01.635003 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:01.635003 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:19:01.635003 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:01.635003 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:19:01.635003 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:19:01.635003 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:01.635003 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:19:01.635003 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:01.635003 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:19:01.635003 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:19:01.635003 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:01.635003 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:19:01.635812 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:01.635812 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:19:01.635812 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:01.635812 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:19:01.635812 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:19:01.635812 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:19:01.635812 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:19:01.635812 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:19:01.635812 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:01.635812 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:19:01.635812 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:19:01.635812 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:01.635812 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:19:01.635812 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:19:01.635812 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:01.637011 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:01.637011 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:19:01.637011 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:01.637011 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:19:01.637011 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:19:01.637011 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:01.637011 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:01.637011 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:01.637011 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:19:01.637011 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:01.637011 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:01.637011 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:01.637011 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:19:01.637011 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:01.637011 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:19:01.637011 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:19:01.637011 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:01.637011 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:19:01.637011 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:01.638133 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:19:01.638133 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:19:01.638133 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:01.638133 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:19:01.638133 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:01.638133 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:19:01.938493 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:19:01.939016 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:19:01.939234 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:19:01.940012 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b027d63e-13a5-445b-84a0-8ae6223e05d5, routes=[], if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:19:01.940249 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:01.941082 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:19:01.941082 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:01.941082 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:19:01.941082 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:19:01.941082 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:01.941082 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:19:01.941082 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:01.941082 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:19:01.942847 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:01.942847 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:19:01.942847 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:01.942847 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:19:01.942847 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:19:01.942847 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:19:01.942847 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:19:01.942847 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:01.942847 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:01.942847 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:01.942847 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:19:01.942847 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:01.943584 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:01.943584 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:01.943584 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:19:01.943584 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:01.943584 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:19:01.943584 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:19:01.943584 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:01.943584 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:19:01.943584 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:01.943584 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:19:01.943584 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:19:01.943584 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:01.943584 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:19:01.944376 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:01.944376 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:19:01.944867 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:01.944867 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:19:01.944867 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:19:01.944867 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:19:01.944867 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:19:01.944867 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:19:01.944867 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:01.944867 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:19:01.944867 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:19:01.944867 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:01.944867 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:19:01.944867 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:19:01.944867 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:01.944867 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:01.944867 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:19:01.944867 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:01.945889 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:19:01.945889 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:19:01.945889 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:01.945889 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:01.945889 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:01.945889 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:19:01.945889 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:01.945889 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:01.945889 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:01.945889 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:19:01.945889 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:01.945889 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:19:01.945889 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:19:01.945889 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:01.945889 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:19:01.945889 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:01.945889 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:19:01.945889 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:19:01.945889 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:01.947019 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:19:01.947019 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:01.947019 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:19:01.954382 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:19:01.954837 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:19:01.955087 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:19:01.955305 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105458) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 15 10:19:01.955913 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:19:01.956258 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:01.957184 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:19:01.957184 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:01.957184 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:19:01.957184 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:19:01.957184 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:01.957184 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:19:01.957184 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:01.957184 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:19:01.958966 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:01.958966 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:19:01.958966 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:19:01.958966 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:19:01.958966 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:01.958966 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:19:01.958966 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:01.958966 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:19:01.958966 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:01.958966 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:19:01.958966 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:01.958966 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:19:01.958966 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:01.958966 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:19:01.958966 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:19:01.959791 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:01.959791 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:19:01.959791 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:01.959791 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:19:01.959791 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:19:01.959791 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:01.959791 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:19:01.959791 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:01.959791 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:19:01.960315 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:01.960315 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:19:01.960315 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:01.960315 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:19:01.960315 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:19:01.960315 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:19:01.960315 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:19:01.960315 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 15 10:19:01.960315 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:19:01.960315 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:19:01.960315 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:01.960315 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:01.960973 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:01.960973 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:19:01.960973 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:01.960973 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:01.960973 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:01.960973 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:19:01.960973 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:01.960973 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:19:01.960973 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:19:01.960973 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:01.960973 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:19:01.960973 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:01.960973 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:19:01.960973 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:19:01.961591 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:01.961591 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:19:01.961591 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:01.961591 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:19:01.961591 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:01.961591 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:19:01.961591 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:19:01.961591 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:19:01.961591 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:19:01.961591 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:19:01.961591 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:01.961591 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:19:01.961591 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:19:01.961591 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:01.962475 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:19:01.962475 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:19:01.962475 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:01.962475 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:01.962475 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:19:01.962475 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:01.962475 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:19:01.962475 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:19:01.962475 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^ May 15 10:19:01.962475 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:19:01.962475 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:19:01.962475 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:01.962475 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:01.962475 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:01.962475 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:19:01.962475 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:01.962475 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:01.962475 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:01.962475 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:19:01.963473 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:01.963473 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:19:01.963473 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:19:01.963473 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:01.963473 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:19:01.963473 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:01.963473 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:19:01.963473 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:19:01.963473 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:01.963473 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:19:01.963473 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:01.963473 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:19:01.967268 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:19:01.967869 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_ha_failover {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:19:01.968046 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:19:01.968372 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:19:01.969179 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:19:01.969790 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:01.970893 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:19:01.970893 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:01.970893 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:19:01.970893 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:19:01.970893 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:01.970893 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:19:01.970893 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:01.970893 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:19:01.971976 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:01.971976 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:19:01.971976 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:19:01.971976 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:19:01.971976 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:01.971976 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:19:01.971976 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:01.971976 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:19:01.971976 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:01.971976 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:19:01.971976 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:01.971976 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:19:01.971976 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:01.971976 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:19:01.971976 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:19:01.972396 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:01.972396 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:19:01.972396 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:01.972396 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:19:01.972396 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:19:01.972396 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:01.972396 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:19:01.972396 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:01.972396 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:19:01.973237 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_ha_failover failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:01.973237 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:19:01.973237 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:01.973237 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:19:01.973237 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:19:01.973237 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:19:01.973237 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 15 10:19:01.973237 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:19:01.973237 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:19:01.973237 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:01.973237 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:01.973767 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:01.973767 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:19:01.973767 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:01.973767 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:01.973767 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:01.973767 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:19:01.973767 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:01.973767 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:19:01.973767 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:19:01.973767 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:01.973767 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:19:01.973767 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:01.973767 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:19:01.973767 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:19:01.974234 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:01.974234 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:19:01.974234 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:01.974234 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:19:01.974845 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:01.974845 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:19:01.974845 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:19:01.974845 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:19:01.974845 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:19:01.974845 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:19:01.974845 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:01.974845 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:19:01.974845 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:19:01.974845 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:01.974845 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:19:01.974845 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:19:01.974845 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:01.974845 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:01.974845 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:19:01.974845 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:01.974845 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:19:01.975612 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 15 10:19:01.975612 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:19:01.975612 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:19:01.975612 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:01.975612 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:01.975612 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:01.975612 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:19:01.975612 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:01.975612 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:01.975612 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:01.975612 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:19:01.975612 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:01.975612 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:19:01.975612 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:19:01.975612 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:01.975612 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:19:01.975612 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:01.975612 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:19:01.975612 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:19:01.976473 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:01.976473 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:19:01.976473 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:01.976473 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:19:01.976473 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:19:01.976473 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:19:01.976473 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:19:01.976473 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:19:01.976903 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:19:01.976903 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:19:01.976903 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:01.976903 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:19:01.976903 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:01.976903 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:19:01.976903 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:19:01.976903 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:01.976903 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:19:01.976903 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:01.976903 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:19:01.977977 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:01.977977 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:19:01.977977 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:01.977977 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:19:01.977977 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:19:01.977977 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:19:01.977977 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:19:01.977977 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:01.977977 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:01.977977 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:01.977977 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:19:01.977977 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:01.978542 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:01.978542 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:01.978542 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:19:01.978542 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:01.978542 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:19:01.978542 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:19:01.978542 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:01.978542 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:19:01.978542 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:01.978542 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:19:01.978542 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:19:01.978542 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:01.978542 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:19:01.979100 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:01.979100 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:19:01.979100 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:01.979100 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:19:01.979100 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:19:01.979100 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:19:01.979100 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:19:01.979100 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:19:01.979100 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:01.979100 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:19:01.979100 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:19:01.979100 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:01.979100 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:19:01.979100 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:19:01.979100 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:01.979775 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:01.979775 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:19:01.979775 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:01.979775 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:19:01.979775 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:19:01.979775 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:01.979775 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:01.979775 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:01.979775 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:19:01.979775 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:01.979775 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:01.979775 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:01.979775 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:19:01.979775 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:01.979775 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:19:01.979775 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:19:01.979775 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:01.979775 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:19:01.979775 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:01.980555 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:19:01.980555 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:19:01.980555 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:01.980555 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:19:01.980555 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:01.980555 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:19:06.393072 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:19:06.393072 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:19:06.393072 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:19:06.394890 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:19:06.438152 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:19:06.438434 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:06.439553 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:19:06.439553 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:06.439553 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:19:06.439553 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:19:06.439553 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:06.439553 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:19:06.439553 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:06.439553 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:19:06.441143 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_fdb_aging_settings failed after 0.046 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:06.441143 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:19:06.441143 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:06.441143 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:19:06.441143 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:19:06.441143 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:19:06.441143 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:19:06.441143 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:06.441143 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:06.441143 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:06.441143 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:19:06.441143 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:06.441529 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:06.441529 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:06.441529 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:19:06.441529 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:06.441529 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:19:06.441529 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:19:06.441529 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:06.441529 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:19:06.441529 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:06.441529 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:19:06.441529 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:19:06.441529 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:06.441529 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:19:06.441884 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:06.441884 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:19:06.442272 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:06.442272 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:19:06.442272 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:19:06.442272 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:19:06.442272 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:19:06.442272 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:19:06.442272 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:06.442272 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:19:06.442272 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:19:06.442272 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:06.442272 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:19:06.442272 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:19:06.442272 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:06.442272 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:06.442272 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:19:06.442272 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:06.442770 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:19:06.442770 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:19:06.442770 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:06.442770 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:06.442770 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:06.442770 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:19:06.442770 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:06.442770 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:06.442770 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:06.442770 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:19:06.442770 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:06.442770 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:19:06.442770 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:19:06.442770 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:06.442770 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:19:06.442770 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:06.442770 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:19:06.442770 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:19:06.442770 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:06.443462 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:19:06.443462 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:06.443462 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:19:06.939005 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:19:06.939580 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:19:06.939825 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:19:06.941206 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b027d63e-13a5-445b-84a0-8ae6223e05d5, routes=[], if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:19:06.941414 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:06.942198 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:19:06.942198 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:06.942198 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:19:06.942198 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:19:06.942198 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:06.942198 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:19:06.942198 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:06.942198 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:19:06.944053 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:06.944053 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:19:06.944053 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:06.944053 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:19:06.944053 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:19:06.944053 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:19:06.944053 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:19:06.944053 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:06.944053 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:06.944053 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:06.944053 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:19:06.944053 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:06.944387 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:06.944387 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:06.944387 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:19:06.944387 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:06.944387 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:19:06.944387 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:19:06.944387 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:06.944387 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:19:06.944387 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:06.944387 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:19:06.944387 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:19:06.944387 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:06.944387 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:19:06.944758 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:06.944758 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:19:06.946542 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:06.946542 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:19:06.946542 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:19:06.946542 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:19:06.946542 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:19:06.946542 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:19:06.946542 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:06.946542 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:19:06.946542 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:19:06.946542 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:06.946542 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:19:06.946542 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:19:06.946542 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:06.946542 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:06.946542 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:19:06.946542 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:06.947261 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:19:06.947261 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:19:06.947261 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:06.947261 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:06.947261 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:06.947261 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:19:06.947261 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:06.947261 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:06.947261 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:06.947261 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:19:06.947261 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:06.947261 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:19:06.947261 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:19:06.947261 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:06.947261 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:19:06.947261 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:06.947261 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:19:06.947261 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:19:06.947261 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:06.948044 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:19:06.948044 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:06.948044 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:19:06.954884 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:19:06.955313 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:19:06.955438 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:19:06.955584 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105458) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 15 10:19:06.956219 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:19:06.956426 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:06.957085 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:19:06.957085 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:06.957085 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:19:06.957085 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:19:06.957085 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:06.957085 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:19:06.957085 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:06.957085 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:19:06.957991 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:06.957991 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:19:06.957991 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:19:06.957991 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:19:06.957991 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:06.957991 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:19:06.957991 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:06.957991 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:19:06.957991 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:06.957991 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:19:06.957991 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:06.957991 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:19:06.957991 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:06.957991 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:19:06.957991 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:19:06.958659 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:06.958659 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:19:06.958659 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:06.958659 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:19:06.958659 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:19:06.958659 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:06.958659 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:19:06.958659 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:06.958659 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:19:06.959150 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:06.959150 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:19:06.959150 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:06.959150 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:19:06.959150 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:19:06.959150 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:19:06.959150 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:19:06.959150 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 15 10:19:06.959150 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:19:06.959150 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:19:06.959150 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:06.959150 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:06.959657 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:06.959657 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:19:06.959657 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:06.959657 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:06.959657 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:06.959657 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:19:06.959657 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:06.959657 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:19:06.959657 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:19:06.959657 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:06.959657 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:19:06.959657 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:06.959657 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:19:06.959657 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:19:06.960596 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:06.960596 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:19:06.960596 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:06.960596 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:19:06.960596 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:06.960596 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:19:06.960596 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:19:06.960596 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:19:06.960596 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:19:06.960596 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:19:06.960596 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:06.960596 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:19:06.960596 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:19:06.960596 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:06.961531 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:19:06.961531 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:19:06.961531 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:06.961531 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:06.961531 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:19:06.961531 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:06.961531 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:19:06.961531 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:19:06.961531 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^ May 15 10:19:06.961531 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:19:06.961531 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:19:06.961531 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:06.961531 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:06.961531 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:06.961531 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:19:06.961531 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:06.961531 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:06.961531 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:06.961531 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:19:06.962905 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:06.962905 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:19:06.962905 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:19:06.962905 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:06.962905 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:19:06.962905 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:06.962905 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:19:06.962905 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:19:06.962905 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:06.962905 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:19:06.962905 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:06.962905 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:19:06.967860 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:19:06.968260 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_ha_failover {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:19:06.968437 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:19:06.968737 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:19:06.969379 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:19:06.969645 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:06.970255 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:19:06.970255 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:06.970255 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:19:06.970255 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:19:06.970255 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:06.970255 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:19:06.970255 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:06.970255 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:19:06.971262 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:06.971262 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:19:06.971262 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:19:06.971262 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:19:06.971262 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:06.971262 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:19:06.971262 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:06.971262 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:19:06.971262 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:06.971262 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:19:06.971262 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:06.971262 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:19:06.971262 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:06.971262 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:19:06.971262 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:19:06.971696 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:06.971696 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:19:06.971696 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:06.971696 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:19:06.971696 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:19:06.971696 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:06.971696 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:19:06.971696 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:06.971696 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:19:06.972202 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:06.972202 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:19:06.972202 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:06.972202 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:19:06.972202 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:19:06.972202 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:19:06.972202 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 15 10:19:06.972202 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:19:06.972202 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:19:06.972202 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:06.972202 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:06.972601 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:06.972601 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:19:06.972601 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:06.972601 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:06.972601 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:06.972601 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:19:06.972601 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:06.972601 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:19:06.972601 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:19:06.972601 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:06.972601 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:19:06.972601 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:06.972601 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:19:06.972601 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:19:06.973000 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:06.973000 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:19:06.973000 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:06.973000 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:19:06.973000 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:06.973000 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:19:06.973000 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:19:06.973000 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:19:06.973000 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:19:06.973000 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:19:06.973000 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:06.973000 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:19:06.973000 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:19:06.973000 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:06.973743 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:19:06.973743 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:19:06.973743 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:06.973743 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:06.973743 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:19:06.973743 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:06.973743 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:19:06.973743 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 15 10:19:06.973743 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:19:06.973743 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:19:06.973743 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:06.973743 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:06.973743 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:06.973743 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:19:06.973743 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:06.973743 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:06.973743 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:06.973743 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:19:06.973743 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:06.974668 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:19:06.974668 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:19:06.974668 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:06.974668 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:19:06.974668 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:06.974668 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:19:06.974668 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:19:06.974668 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:06.974668 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:19:06.974668 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:06.974668 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:19:06.975369 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:19:06.975654 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:19:06.975802 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:19:06.975975 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:19:06.976697 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:19:06.976975 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:19:06.977096 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:06.977631 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:19:06.977631 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:06.977631 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:19:06.977631 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:19:06.977631 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:06.977631 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:19:06.977631 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:06.977631 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:19:06.978578 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:06.978578 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:19:06.978578 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:06.978578 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:19:06.978578 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:19:06.978578 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:19:06.978578 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:19:06.978578 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:06.978578 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:06.978578 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:06.978578 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:19:06.978578 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:06.978948 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:06.978948 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:06.978948 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:19:06.978948 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:06.978948 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:19:06.978948 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:19:06.978948 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:06.978948 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:19:06.978948 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:06.978948 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:19:06.978948 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:19:06.978948 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:06.978948 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:19:06.979291 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:06.979291 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:19:06.979599 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:06.979599 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:19:06.979599 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:19:06.979599 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:19:06.979599 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:19:06.979599 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:19:06.979599 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:06.979599 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:19:06.979599 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:19:06.979599 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:06.979599 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:19:06.979599 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:19:06.979599 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:06.979599 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:06.979599 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:19:06.979599 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:06.980052 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:19:06.980052 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:19:06.980052 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:06.980052 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:06.980052 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:06.980052 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:19:06.980052 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:06.980052 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:06.980052 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:06.980052 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:19:06.980052 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:06.980052 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:19:06.980052 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:19:06.980052 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:06.980052 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:19:06.980052 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:06.980052 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:19:06.980052 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:19:06.980052 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:06.980762 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:19:06.980762 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:06.980762 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:19:11.393990 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:19:11.394960 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:19:11.394960 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:19:11.398699 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:19:11.442513 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:19:11.442866 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:11.443774 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:19:11.443774 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:11.443774 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:19:11.443774 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:19:11.443774 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:11.443774 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:19:11.443774 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:11.443774 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:19:11.445080 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_fdb_aging_settings failed after 0.049 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:11.445080 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:19:11.445080 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:11.445080 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:19:11.445080 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:19:11.445080 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:19:11.445080 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:19:11.445080 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:11.445080 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:11.445080 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:11.445080 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:19:11.445080 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:11.445656 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:11.445656 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:11.445656 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:19:11.445656 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:11.445656 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:19:11.445656 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:19:11.445656 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:11.445656 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:19:11.445656 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:11.445656 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:19:11.445656 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:19:11.445656 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:11.445656 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:19:11.446207 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:11.446207 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:19:11.446207 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:11.446207 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:19:11.446207 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:19:11.446207 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:19:11.446207 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:19:11.446207 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:19:11.446207 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:11.446207 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:19:11.446207 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:19:11.446207 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:11.446207 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:19:11.446207 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:19:11.446207 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:11.446869 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:11.446869 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:19:11.446869 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:11.446869 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:19:11.446869 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:19:11.446869 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:11.446869 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:11.446869 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:11.446869 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:19:11.446869 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:11.446869 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:11.446869 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:11.446869 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:19:11.446869 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:11.446869 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:19:11.446869 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:19:11.446869 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:11.446869 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:19:11.446869 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:11.447663 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:19:11.447663 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:19:11.447663 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:11.447663 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:19:11.447663 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:11.447663 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:19:11.940022 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:19:11.940883 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:19:11.941096 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:19:11.942306 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b027d63e-13a5-445b-84a0-8ae6223e05d5, routes=[], if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:19:11.942665 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:11.943644 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:19:11.943644 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:11.943644 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:19:11.943644 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:19:11.943644 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:11.943644 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:19:11.943644 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:11.943644 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:19:11.945660 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:11.945660 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:19:11.945660 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:11.945660 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:19:11.945660 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:19:11.945660 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:19:11.945660 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:19:11.945660 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:11.945660 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:11.945660 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:11.945660 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:19:11.945660 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:11.946354 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:11.946354 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:11.946354 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:19:11.946354 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:11.946354 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:19:11.946354 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:19:11.946354 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:11.946354 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:19:11.946354 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:11.946354 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:19:11.946354 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:19:11.946354 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:11.946354 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:19:11.946946 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:11.946946 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:19:11.947655 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:11.947655 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:19:11.947655 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:19:11.947655 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:19:11.947655 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:19:11.947655 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:19:11.947655 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:11.947655 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:19:11.947655 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:19:11.947655 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:11.947655 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:19:11.947655 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:19:11.947655 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:11.947655 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:11.947655 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:19:11.947655 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:11.948649 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:19:11.948649 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:19:11.948649 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:11.948649 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:11.948649 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:11.948649 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:19:11.948649 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:11.948649 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:11.948649 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:11.948649 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:19:11.948649 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:11.948649 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:19:11.948649 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:19:11.948649 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:11.948649 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:19:11.948649 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:11.948649 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:19:11.948649 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:19:11.948649 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:11.949884 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:19:11.949884 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:11.949884 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:19:11.955567 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:19:11.956015 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:19:11.956162 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:19:11.956292 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105458) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 15 10:19:11.956990 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:19:11.957309 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:11.957948 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:19:11.957948 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:11.957948 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:19:11.957948 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:19:11.957948 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:11.957948 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:19:11.957948 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:11.957948 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:19:11.958789 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:11.958789 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:19:11.958789 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:19:11.958789 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:19:11.958789 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:11.958789 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:19:11.958789 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:11.958789 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:19:11.958789 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:11.958789 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:19:11.958789 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:11.958789 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:19:11.958789 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:11.958789 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:19:11.958789 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:19:11.959220 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:11.959220 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:19:11.959220 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:11.959220 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:19:11.959220 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:19:11.959220 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:11.959220 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:19:11.959220 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:11.959220 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:19:11.959760 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:11.959760 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:19:11.959760 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:11.959760 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:19:11.959760 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:19:11.959760 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:19:11.959760 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:19:11.959760 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 15 10:19:11.959760 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:19:11.959760 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:19:11.959760 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:11.959760 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:11.960124 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:11.960124 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:19:11.960124 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:11.960124 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:11.960124 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:11.960124 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:19:11.960124 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:11.960124 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:19:11.960124 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:19:11.960124 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:11.960124 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:19:11.960124 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:11.960124 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:19:11.960124 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:19:11.960475 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:11.960475 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:19:11.960475 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:11.960475 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:19:11.961605 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:11.961605 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:19:11.961605 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:19:11.961605 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:19:11.961605 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:19:11.961605 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:19:11.961605 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:11.961605 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:19:11.961605 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:19:11.961605 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:11.961605 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:19:11.961605 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:19:11.961605 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:11.961605 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:11.961605 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:19:11.961605 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:11.962120 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:19:11.962120 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:19:11.962120 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^ May 15 10:19:11.962120 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:19:11.962120 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:19:11.962120 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:11.962120 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:11.962120 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:11.962120 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:19:11.962120 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:11.962120 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:11.962120 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:11.962120 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:19:11.962120 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:11.962120 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:19:11.962120 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:19:11.962120 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:11.962120 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:19:11.962627 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:11.962627 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:19:11.962627 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:19:11.962627 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:11.962627 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:19:11.962627 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:11.962627 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:19:11.968419 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:19:11.968855 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_ha_failover {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:19:11.968990 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:19:11.969148 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:19:11.970039 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:19:11.970511 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:11.971444 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:19:11.971444 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:11.971444 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:19:11.971444 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:19:11.971444 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:11.971444 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:19:11.971444 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:11.971444 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:19:11.972833 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:11.972833 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:19:11.972833 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:19:11.972833 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:19:11.972833 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:11.972833 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:19:11.972833 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:11.972833 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:19:11.972833 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:11.972833 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:19:11.972833 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:11.972833 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:19:11.972833 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:11.972833 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:19:11.972833 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:19:11.973491 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:11.973491 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:19:11.973491 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:11.973491 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:19:11.973491 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:19:11.973491 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:11.973491 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:19:11.973491 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:11.973491 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:19:11.974544 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_ha_failover failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:11.974544 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:19:11.974544 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:11.974544 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:19:11.974544 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:19:11.974544 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:19:11.974544 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 15 10:19:11.974544 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:19:11.974544 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:19:11.974544 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:11.974544 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:11.975154 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:11.975154 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:19:11.975154 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:11.975154 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:11.975154 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:11.975154 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:19:11.975154 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:11.975154 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:19:11.975154 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:19:11.975154 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:11.975154 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:19:11.975154 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:11.975154 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:19:11.975154 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:19:11.975862 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:11.975862 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:19:11.975862 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:11.975862 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:19:11.978156 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:11.978156 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:19:11.978156 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:19:11.978156 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:19:11.978156 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:19:11.978156 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:19:11.978156 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:11.978156 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:19:11.978156 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:19:11.978156 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:11.978156 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:19:11.978156 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:19:11.978156 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:11.978156 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:11.978156 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:19:11.978156 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:11.978156 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:19:11.978962 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 15 10:19:11.978962 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:19:11.978962 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:19:11.978962 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:11.978962 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:11.978962 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:11.978962 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:19:11.978962 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:11.978962 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:11.978962 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:11.978962 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:19:11.978962 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:11.978962 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:19:11.978962 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:19:11.978962 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:11.978962 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:19:11.978962 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:11.978962 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:19:11.978962 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:19:11.979458 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:11.979458 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:19:11.979458 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:11.979458 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:19:11.980068 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:19:11.980476 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:19:11.980582 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:19:11.980773 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:19:11.981575 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:19:11.982066 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:19:11.982221 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:11.983074 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:19:11.983074 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:11.983074 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:19:11.983074 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:19:11.983074 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:11.983074 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:19:11.983074 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:11.983074 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:19:11.983702 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:11.983702 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:19:11.983702 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:11.983702 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:19:11.983702 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:19:11.983702 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:19:11.983702 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:19:11.983702 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:11.983702 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:11.983702 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:11.983702 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:19:11.983702 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:11.984263 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:11.984263 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:11.984263 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:19:11.984263 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:11.984263 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:19:11.984263 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:19:11.984263 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:11.984263 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:19:11.984263 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:11.984263 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:19:11.984263 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:19:11.984263 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:11.984263 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:19:11.984882 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:11.984882 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:19:11.984882 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:11.984882 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:19:11.984882 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:19:11.984882 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:19:11.984882 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:19:11.984882 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:19:11.984882 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:11.984882 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:19:11.984882 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:19:11.984882 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:11.984882 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:19:11.984882 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:19:11.984882 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:11.985824 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:11.985824 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:19:11.985824 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:11.985824 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:19:11.985824 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:19:11.985824 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:11.985824 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:11.985824 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:11.985824 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:19:11.985824 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:11.985824 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:11.985824 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:11.985824 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:19:11.985824 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:11.985824 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:19:11.985824 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:19:11.985824 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:11.985824 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:19:11.985824 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:11.986821 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:19:11.986821 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:19:11.986821 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:11.986821 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:19:11.986821 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:11.986821 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:19:16.394583 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:19:16.395197 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:19:16.395494 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:19:16.397124 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:19:16.437759 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:19:16.437759 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:16.438034 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:19:16.438034 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:16.438034 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:19:16.438034 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:19:16.438034 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:16.438034 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:19:16.438034 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:16.438034 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:19:16.438964 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_fdb_aging_settings failed after 0.043 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:16.438964 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:19:16.438964 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:16.438964 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:19:16.438964 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:19:16.438964 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:19:16.438964 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:19:16.438964 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:16.438964 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:16.438964 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:16.438964 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:19:16.438964 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:16.439407 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:16.439407 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:16.439407 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:19:16.439407 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:16.439407 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:19:16.439407 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:19:16.439407 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:16.439407 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:19:16.439407 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:16.439407 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:19:16.439407 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:19:16.439407 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:16.439407 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:19:16.439770 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:16.439770 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:19:16.440056 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:16.440056 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:19:16.440056 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:19:16.440056 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:19:16.440056 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:19:16.440056 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:19:16.440056 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:16.440056 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:19:16.440056 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:19:16.440056 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:16.440056 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:19:16.440056 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:19:16.440056 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:16.440056 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:16.440056 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:19:16.440056 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:16.440528 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:19:16.440528 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:19:16.440528 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:16.440528 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:16.440528 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:16.440528 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:19:16.440528 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:16.440528 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:16.440528 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:16.440528 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:19:16.440528 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:16.440528 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:19:16.440528 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:19:16.440528 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:16.440528 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:19:16.440528 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:16.440528 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:19:16.440528 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:19:16.440528 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:16.441185 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:19:16.441185 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:16.441185 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:19:16.940929 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:19:16.941307 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:19:16.941631 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:19:16.942502 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b027d63e-13a5-445b-84a0-8ae6223e05d5, routes=[], if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:19:16.942930 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:16.943830 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:19:16.943830 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:16.943830 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:19:16.943830 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:19:16.943830 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:16.943830 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:19:16.943830 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:16.943830 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:19:16.945005 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:16.945005 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:19:16.945005 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:16.945005 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:19:16.945005 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:19:16.945005 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:19:16.945005 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:19:16.945005 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:16.945005 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:16.945005 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:16.945005 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:19:16.945005 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:16.945433 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:16.945433 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:16.945433 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:19:16.945433 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:16.945433 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:19:16.945433 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:19:16.945433 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:16.945433 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:19:16.945433 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:16.945433 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:19:16.945433 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:19:16.945433 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:16.945433 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:19:16.945883 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:16.945883 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:19:16.946527 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:16.946527 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:19:16.946527 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:19:16.946527 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:19:16.946527 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:19:16.946527 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:19:16.946527 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:16.946527 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:19:16.946527 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:19:16.946527 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:16.946527 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:19:16.946527 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:19:16.946527 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:16.946527 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:16.946527 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:19:16.946527 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:16.946998 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:19:16.946998 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:19:16.946998 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:16.946998 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:16.946998 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:16.946998 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:19:16.946998 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:16.946998 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:16.946998 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:16.946998 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:19:16.946998 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:16.946998 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:19:16.946998 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:19:16.946998 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:16.946998 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:19:16.946998 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:16.946998 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:19:16.946998 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:19:16.946998 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:16.947490 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:19:16.947490 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:16.947490 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:19:16.956164 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:19:16.956501 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:19:16.956700 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:19:16.956930 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105458) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 15 10:19:16.957617 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:19:16.958108 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:16.959279 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:19:16.959279 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:16.959279 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:19:16.959279 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:19:16.959279 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:16.959279 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:19:16.959279 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:16.959279 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:19:16.960897 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:16.960897 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:19:16.960897 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:19:16.960897 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:19:16.960897 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:16.960897 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:19:16.960897 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:16.960897 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:19:16.960897 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:16.960897 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:19:16.960897 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:16.960897 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:19:16.960897 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:16.960897 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:19:16.960897 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:19:16.961339 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:16.961339 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:19:16.961339 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:16.961339 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:19:16.961339 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:19:16.961339 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:16.961339 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:19:16.961339 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:16.961339 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:19:16.962793 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:16.962793 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:19:16.962793 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:16.962793 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:19:16.962793 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:19:16.962793 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:19:16.962793 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:19:16.962793 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 15 10:19:16.962793 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:19:16.962793 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:19:16.962793 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:16.962793 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:16.963163 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:16.963163 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:19:16.963163 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:16.963163 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:16.963163 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:16.963163 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:19:16.963163 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:16.963163 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:19:16.963163 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:19:16.963163 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:16.963163 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:19:16.963163 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:16.963163 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:19:16.963163 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:19:16.963552 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:16.963552 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:19:16.963552 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:16.963552 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:19:16.964800 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:16.964800 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:19:16.964800 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:19:16.964800 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:19:16.964800 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:19:16.964800 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:19:16.964800 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:16.964800 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:19:16.964800 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:19:16.964800 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:16.964800 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:19:16.964800 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:19:16.964800 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:16.964800 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:16.964800 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:19:16.964800 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:16.965241 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:19:16.965241 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:19:16.965241 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^ May 15 10:19:16.965241 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:19:16.965241 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:19:16.965241 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:16.965241 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:16.965241 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:16.965241 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:19:16.965241 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:16.965241 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:16.965241 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:16.965241 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:19:16.965241 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:16.965241 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:19:16.965241 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:19:16.965241 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:16.965241 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:19:16.965733 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:16.965733 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:19:16.965733 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:19:16.965733 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:16.965733 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:19:16.965733 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:16.965733 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:19:16.969001 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:19:16.969323 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_ha_failover {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:19:16.969512 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:19:16.969756 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:19:16.970227 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:19:16.970497 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:16.971081 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:19:16.971081 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:16.971081 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:19:16.971081 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:19:16.971081 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:16.971081 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:19:16.971081 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:16.971081 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:19:16.972652 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:16.972652 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:19:16.972652 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:19:16.972652 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:19:16.972652 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:16.972652 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:19:16.972652 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:16.972652 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:19:16.972652 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:16.972652 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:19:16.972652 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:16.972652 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:19:16.972652 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:16.972652 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:19:16.972652 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:19:16.973656 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:16.973656 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:19:16.973656 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:16.973656 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:19:16.973656 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:19:16.973656 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:16.973656 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:19:16.973656 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:16.973656 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:19:16.973656 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:16.973656 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:19:16.973656 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:16.973656 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:19:16.973656 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:19:16.974390 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:19:16.974390 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 15 10:19:16.974390 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:19:16.974390 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:19:16.974390 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:16.974390 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:16.974390 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:16.974390 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:19:16.974390 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:16.974390 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:16.974390 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:16.974390 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:19:16.974390 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:16.975091 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:19:16.975091 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:19:16.975091 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:16.975091 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:19:16.975091 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:16.975091 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:19:16.975091 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:19:16.975091 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:16.975091 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:19:16.975091 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:16.975091 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:19:16.975593 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:16.975593 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:19:16.975593 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:19:16.975593 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:19:16.975593 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:19:16.975593 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:19:16.975593 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:16.975593 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:19:16.975593 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:19:16.975593 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:16.975593 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:19:16.975593 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:19:16.975593 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:16.975593 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:16.975593 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:19:16.975593 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:16.975593 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:19:16.976592 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 15 10:19:16.976592 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:19:16.976592 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:19:16.976592 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:16.976592 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:16.976592 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:16.976592 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:19:16.976592 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:16.976592 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:16.976592 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:16.976592 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:19:16.976592 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:16.976592 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:19:16.976592 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:19:16.976592 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:16.976592 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:19:16.976592 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:16.976592 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:19:16.976592 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:19:16.977513 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:16.977513 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:19:16.977513 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:16.977513 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:19:16.980642 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:19:16.981005 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:19:16.981208 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:19:16.981433 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:19:16.981857 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:19:16.982127 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:19:16.982238 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:16.982866 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:19:16.982866 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:16.982866 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:19:16.982866 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:19:16.982866 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:16.982866 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:19:16.982866 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:16.982866 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:19:16.984054 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:16.984054 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:19:16.984054 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:16.984054 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:19:16.984054 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:19:16.984054 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:19:16.984054 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:19:16.984054 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:16.984054 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:16.984054 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:16.984054 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:19:16.984054 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:16.984514 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:16.984514 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:16.984514 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:19:16.984514 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:16.984514 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:19:16.984514 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:19:16.984514 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:16.984514 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:19:16.984514 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:16.984514 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:19:16.984514 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:19:16.984514 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:16.984514 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:19:16.984895 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:16.984895 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:19:16.985192 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:16.985192 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:19:16.985192 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:19:16.985192 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:19:16.985192 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:19:16.985192 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:19:16.985192 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:16.985192 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:19:16.985192 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:19:16.985192 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:16.985192 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:19:16.985192 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:19:16.985192 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:16.985192 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:16.985192 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:19:16.985192 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:16.985842 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:19:16.985842 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:19:16.985842 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:16.985842 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:16.985842 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:16.985842 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:19:16.985842 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:16.985842 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:16.985842 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:16.985842 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:19:16.985842 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:16.985842 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:19:16.985842 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:19:16.985842 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:16.985842 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:19:16.985842 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:16.985842 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:19:16.985842 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:19:16.985842 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:16.986664 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:19:16.986664 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:16.986664 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:19:21.395395 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:19:21.396110 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:19:21.396366 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:19:21.397841 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:19:21.440653 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:19:21.441109 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:21.442113 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:19:21.442113 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:21.442113 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:19:21.442113 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:19:21.442113 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:21.442113 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:19:21.442113 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:21.442113 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:19:21.443209 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_fdb_aging_settings failed after 0.046 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:21.443209 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:19:21.443209 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:21.443209 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:19:21.443209 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:19:21.443209 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:19:21.443209 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:19:21.443209 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:21.443209 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:21.443209 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:21.443209 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:19:21.443209 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:21.443668 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:21.443668 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:21.443668 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:19:21.443668 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:21.443668 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:19:21.443668 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:19:21.443668 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:21.443668 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:19:21.443668 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:21.443668 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:19:21.443668 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:19:21.443668 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:21.443668 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:19:21.444086 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:21.444086 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:19:21.444438 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:21.444438 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:19:21.444438 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:19:21.444438 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:19:21.444438 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:19:21.444438 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:19:21.444438 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:21.444438 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:19:21.444438 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:19:21.444438 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:21.444438 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:19:21.444438 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:19:21.444438 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:21.444438 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:21.444438 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:19:21.444438 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:21.444879 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:19:21.444879 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:19:21.444879 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:21.444879 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:21.444879 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:21.444879 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:19:21.444879 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:21.444879 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:21.444879 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:21.444879 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:19:21.444879 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:21.444879 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:19:21.444879 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:19:21.444879 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:21.444879 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:19:21.444879 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:21.444879 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:19:21.444879 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:19:21.444879 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:21.445396 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:19:21.445396 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:21.445396 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:19:21.941514 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:19:21.942180 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:19:21.942258 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:19:21.943208 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b027d63e-13a5-445b-84a0-8ae6223e05d5, routes=[], if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:19:21.943647 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:21.944606 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:19:21.944606 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:21.944606 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:19:21.944606 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:19:21.944606 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:21.944606 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:19:21.944606 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:21.944606 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:19:21.945532 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:21.945532 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:19:21.945532 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:21.945532 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:19:21.945532 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:19:21.945532 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:19:21.945532 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:19:21.945532 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:21.945532 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:21.945532 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:21.945532 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:19:21.945532 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:21.945986 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:21.945986 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:21.945986 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:19:21.945986 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:21.945986 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:19:21.945986 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:19:21.945986 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:21.945986 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:19:21.945986 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:21.945986 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:19:21.945986 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:19:21.945986 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:21.945986 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:19:21.946419 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:21.946419 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:19:21.946642 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:21.946642 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:19:21.946642 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:19:21.946642 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:19:21.946642 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:19:21.946642 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:19:21.946642 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:21.946642 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:19:21.946642 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:19:21.946642 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:21.946642 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:19:21.946642 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:19:21.946642 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:21.946642 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:21.946642 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:19:21.946642 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:21.947322 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:19:21.947322 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:19:21.947322 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:21.947322 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:21.947322 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:21.947322 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:19:21.947322 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:21.947322 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:21.947322 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:21.947322 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:19:21.947322 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:21.947322 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:19:21.947322 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:19:21.947322 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:21.947322 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:19:21.947322 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:21.947322 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:19:21.947322 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:19:21.947322 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:21.948043 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:19:21.948043 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:21.948043 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:19:21.956883 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:19:21.957239 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:19:21.957377 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:19:21.957531 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105458) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 15 10:19:21.958305 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:19:21.958593 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:21.959452 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:19:21.959452 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:21.959452 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:19:21.959452 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:19:21.959452 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:21.959452 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:19:21.959452 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:21.959452 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:19:21.961114 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:21.961114 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:19:21.961114 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:19:21.961114 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:19:21.961114 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:21.961114 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:19:21.961114 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:21.961114 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:19:21.961114 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:21.961114 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:19:21.961114 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:21.961114 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:19:21.961114 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:21.961114 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:19:21.961114 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:19:21.962083 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:21.962083 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:19:21.962083 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:21.962083 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:19:21.962083 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:19:21.962083 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:21.962083 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:19:21.962083 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:21.962083 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:19:21.962804 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:21.962804 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:19:21.962804 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:21.962804 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:19:21.962804 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:19:21.962804 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:19:21.962804 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:19:21.962804 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 15 10:19:21.962804 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:19:21.962804 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:19:21.962804 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:21.962804 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:21.963191 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:21.963191 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:19:21.963191 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:21.963191 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:21.963191 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:21.963191 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:19:21.963191 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:21.963191 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:19:21.963191 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:19:21.963191 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:21.963191 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:19:21.963191 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:21.963191 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:19:21.963191 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:19:21.963531 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:21.963531 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:19:21.963531 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:21.963531 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:19:21.965503 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:21.965503 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:19:21.965503 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:19:21.965503 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:19:21.965503 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:19:21.965503 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:19:21.965503 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:21.965503 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:19:21.965503 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:19:21.965503 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:21.965503 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:19:21.965503 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:19:21.965503 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:21.965503 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:21.965503 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:19:21.965503 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:21.966042 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:19:21.966042 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:19:21.966042 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^ May 15 10:19:21.966042 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:19:21.966042 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:19:21.966042 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:21.966042 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:21.966042 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:21.966042 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:19:21.966042 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:21.966042 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:21.966042 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:21.966042 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:19:21.966042 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:21.966042 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:19:21.966042 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:19:21.966042 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:21.966042 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:19:21.966554 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:21.966554 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:19:21.966554 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:19:21.966554 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:21.966554 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:19:21.966554 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:21.966554 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:19:21.969508 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:19:21.969754 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_ha_failover {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:19:21.969871 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:19:21.970002 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:19:21.970818 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:19:21.971301 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:21.972159 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:19:21.972159 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:21.972159 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:19:21.972159 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:19:21.972159 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:21.972159 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:19:21.972159 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:21.972159 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:19:21.972930 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:21.972930 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:19:21.972930 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:19:21.972930 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:19:21.972930 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:21.972930 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:19:21.972930 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:21.972930 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:19:21.972930 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:21.972930 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:19:21.972930 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:21.972930 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:19:21.972930 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:21.972930 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:19:21.972930 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:19:21.975149 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:21.975149 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:19:21.975149 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:21.975149 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:19:21.975149 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:19:21.975149 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:21.975149 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:19:21.975149 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:21.975149 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:19:21.975149 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:21.975149 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:19:21.975149 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:21.975149 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:19:21.975149 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:19:21.975573 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:19:21.975573 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 15 10:19:21.975573 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:19:21.975573 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:19:21.975573 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:21.975573 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:21.975573 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:21.975573 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:19:21.975573 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:21.975573 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:21.975573 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:21.975573 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:19:21.975573 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:21.976038 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:19:21.976038 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:19:21.976038 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:21.976038 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:19:21.976038 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:21.976038 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:19:21.976038 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:19:21.976038 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:21.976038 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:19:21.976038 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:21.976038 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:19:21.976038 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:21.976381 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:19:21.976381 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:19:21.976381 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:19:21.976381 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:19:21.976381 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:19:21.976381 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:21.976381 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:19:21.976381 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:19:21.976381 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:21.976381 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:19:21.976381 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:19:21.976381 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:21.976381 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:21.976381 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:19:21.976381 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:21.976381 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:19:21.976381 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 15 10:19:21.976381 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:19:21.976381 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:19:21.977059 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:21.977059 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:21.977059 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:21.977059 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:19:21.977059 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:21.977059 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:21.977059 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:21.977059 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:19:21.977059 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:21.977059 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:19:21.977059 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:19:21.977059 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:21.977059 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:19:21.977059 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:21.977059 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:19:21.977059 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:19:21.977059 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:21.977059 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:19:21.977059 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:21.977059 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:19:21.981245 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:19:21.981666 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:19:21.981895 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:19:21.982147 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:19:21.982713 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:19:21.983166 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:19:21.983404 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:21.984429 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:19:21.984429 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:21.984429 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:19:21.984429 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:19:21.984429 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:21.984429 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:19:21.984429 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:21.984429 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:19:21.986123 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:21.986123 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:19:21.986123 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:21.986123 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:19:21.986123 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:19:21.986123 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:19:21.986123 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:19:21.986123 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:21.986123 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:21.986123 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:21.986123 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:19:21.986123 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:21.986751 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:21.986751 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:21.986751 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:19:21.986751 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:21.986751 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:19:21.986751 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:19:21.986751 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:21.986751 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:19:21.986751 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:21.986751 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:19:21.986751 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:19:21.986751 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:21.986751 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:19:21.987318 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:21.987318 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:19:21.989748 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:21.989748 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:19:21.989748 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:19:21.989748 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:19:21.989748 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:19:21.989748 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:19:21.989748 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:21.989748 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:19:21.989748 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:19:21.989748 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:21.989748 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:19:21.989748 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:19:21.989748 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:21.989748 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:21.989748 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:19:21.989748 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:21.990218 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:19:21.990218 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:19:21.990218 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:21.990218 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:21.990218 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:21.990218 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:19:21.990218 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:21.990218 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:21.990218 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:21.990218 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:19:21.990218 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:21.990218 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:19:21.990218 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:19:21.990218 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:21.990218 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:19:21.990218 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:21.990218 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:19:21.990218 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:19:21.990218 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:21.990565 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:19:21.990565 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:21.990565 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:19:26.396320 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:19:26.397086 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:19:26.397288 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:19:26.398619 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:19:26.434613 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:19:26.434869 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:26.435500 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:19:26.435500 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:26.435500 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:19:26.435500 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:19:26.435500 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:26.435500 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:19:26.435500 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:26.435500 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:19:26.436370 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_fdb_aging_settings failed after 0.038 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:26.436370 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:19:26.436370 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:26.436370 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:19:26.436370 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:19:26.436370 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:19:26.436370 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:19:26.436370 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:26.436370 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:26.436370 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:26.436370 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:19:26.436370 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:26.437021 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:26.437021 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:26.437021 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:19:26.437021 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:26.437021 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:19:26.437021 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:19:26.437021 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:26.437021 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:19:26.437021 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:26.437021 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:19:26.437021 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:19:26.437021 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:26.437021 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:19:26.437814 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:26.437814 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:19:26.437814 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:26.437814 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:19:26.437814 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:19:26.437814 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:19:26.437814 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:19:26.437814 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:19:26.437814 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:26.437814 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:19:26.437814 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:19:26.437814 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:26.437814 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:19:26.437814 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:19:26.437814 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:26.438745 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:26.438745 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:19:26.438745 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:26.438745 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:19:26.438745 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:19:26.438745 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:26.438745 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:26.438745 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:26.438745 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:19:26.438745 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:26.438745 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:26.438745 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:26.438745 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:19:26.438745 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:26.438745 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:19:26.438745 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:19:26.438745 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:26.438745 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:19:26.438745 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:26.439594 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:19:26.439594 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:19:26.439594 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:26.439594 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:19:26.439594 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:26.439594 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:19:26.943142 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:19:26.943938 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:19:26.944058 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:19:26.945464 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b027d63e-13a5-445b-84a0-8ae6223e05d5, routes=[], if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:19:26.945716 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:26.946287 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:19:26.946287 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:26.946287 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:19:26.946287 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:19:26.946287 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:26.946287 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:19:26.946287 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:26.946287 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:19:26.947414 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:26.947414 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:19:26.947414 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:26.947414 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:19:26.947414 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:19:26.947414 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:19:26.947414 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:19:26.947414 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:26.947414 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:26.947414 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:26.947414 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:19:26.947414 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:26.947925 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:26.947925 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:26.947925 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:19:26.947925 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:26.947925 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:19:26.947925 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:19:26.947925 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:26.947925 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:19:26.947925 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:26.947925 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:19:26.947925 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:19:26.947925 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:26.947925 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:19:26.948317 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:26.948317 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:19:26.948711 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:26.948711 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:19:26.948711 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:19:26.948711 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:19:26.948711 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:19:26.948711 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:19:26.948711 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:26.948711 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:19:26.948711 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:19:26.948711 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:26.948711 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:19:26.948711 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:19:26.948711 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:26.948711 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:26.948711 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:19:26.948711 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:26.949306 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:19:26.949306 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:19:26.949306 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:26.949306 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:26.949306 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:26.949306 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:19:26.949306 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:26.949306 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:26.949306 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:26.949306 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:19:26.949306 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:26.949306 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:19:26.949306 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:19:26.949306 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:26.949306 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:19:26.949306 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:26.949306 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:19:26.949306 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:19:26.949306 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:26.949908 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:19:26.949908 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:26.949908 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:19:26.957422 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:19:26.957789 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:19:26.957946 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:19:26.958123 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105458) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 15 10:19:26.958845 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:19:26.959086 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:26.959797 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:19:26.959797 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:26.959797 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:19:26.959797 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:19:26.959797 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:26.959797 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:19:26.959797 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:26.959797 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:19:26.960885 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:26.960885 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:19:26.960885 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:19:26.960885 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:19:26.960885 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:26.960885 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:19:26.960885 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:26.960885 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:19:26.960885 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:26.960885 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:19:26.960885 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:26.960885 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:19:26.960885 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:26.960885 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:19:26.960885 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:19:26.961554 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:26.961554 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:19:26.961554 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:26.961554 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:19:26.961554 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:19:26.961554 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:26.961554 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:19:26.961554 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:26.961554 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:19:26.962093 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:26.962093 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:19:26.962093 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:26.962093 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:19:26.962093 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:19:26.962093 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:19:26.962093 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:19:26.962093 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 15 10:19:26.962093 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:19:26.962093 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:19:26.962093 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:26.962093 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:26.962627 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:26.962627 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:19:26.962627 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:26.962627 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:26.962627 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:26.962627 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:19:26.962627 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:26.962627 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:19:26.962627 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:19:26.962627 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:26.962627 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:19:26.962627 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:26.962627 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:19:26.962627 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:19:26.964066 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:26.964066 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:19:26.964066 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:26.964066 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:19:26.964710 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:26.964710 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:19:26.964710 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:19:26.964710 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:19:26.964710 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:19:26.964710 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:19:26.964710 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:26.964710 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:19:26.964710 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:19:26.964710 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:26.964710 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:19:26.964710 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:19:26.964710 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:26.964710 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:26.964710 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:19:26.964710 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:26.965602 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:19:26.965602 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:19:26.965602 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^ May 15 10:19:26.965602 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:19:26.965602 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:19:26.965602 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:26.965602 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:26.965602 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:26.965602 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:19:26.965602 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:26.965602 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:26.965602 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:26.965602 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:19:26.965602 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:26.965602 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:19:26.965602 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:19:26.965602 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:26.965602 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:19:26.966121 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:26.966121 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:19:26.966121 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:19:26.966121 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:26.966121 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:19:26.966121 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:26.966121 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:19:26.969896 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:19:26.970138 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_ha_failover {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:19:26.970295 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:19:26.970474 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:19:26.971291 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:19:26.971686 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:26.972412 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:19:26.972412 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:26.972412 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:19:26.972412 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:19:26.972412 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:26.972412 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:19:26.972412 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:26.972412 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:19:26.973407 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:26.973407 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:19:26.973407 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:19:26.973407 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:19:26.973407 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:26.973407 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:19:26.973407 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:26.973407 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:19:26.973407 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:26.973407 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:19:26.973407 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:26.973407 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:19:26.973407 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:26.973407 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:19:26.973407 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:19:26.974056 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:26.974056 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:19:26.974056 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:26.974056 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:19:26.974056 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:19:26.974056 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:26.974056 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:19:26.974056 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:26.974056 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:19:26.974660 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:26.974660 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:19:26.974660 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:26.974660 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:19:26.974660 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:19:26.974660 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:19:26.974660 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 15 10:19:26.974660 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:19:26.974660 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:19:26.974660 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:26.974660 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:26.975963 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:26.975963 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:19:26.975963 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:26.975963 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:26.975963 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:26.975963 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:19:26.975963 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:26.975963 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:19:26.975963 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:19:26.975963 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:26.975963 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:19:26.975963 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:26.975963 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:19:26.975963 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:19:26.976462 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:26.976462 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:19:26.976462 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:26.976462 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:19:26.976462 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:26.976462 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:19:26.976462 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:19:26.976462 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:19:26.976462 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:19:26.976462 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:19:26.976462 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:26.976462 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:19:26.976462 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:19:26.976462 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:26.977004 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:19:26.977004 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:19:26.977004 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:26.977004 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:26.977004 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:19:26.977004 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:26.977004 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:19:26.977004 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 15 10:19:26.977004 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:19:26.977004 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:19:26.977004 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:26.977004 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:26.977004 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:26.977004 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:19:26.977004 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:26.977004 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:26.977004 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:26.977004 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:19:26.977004 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:26.977592 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:19:26.977592 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:19:26.977592 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:26.977592 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:19:26.977592 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:26.977592 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:19:26.977592 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:19:26.977592 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:26.977592 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:19:26.977592 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:26.977592 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:19:26.981721 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:19:26.981908 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:19:26.982053 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:19:26.982242 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:19:26.983005 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:19:26.983427 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:19:26.983672 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:26.984599 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:19:26.984599 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:26.984599 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:19:26.984599 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:19:26.984599 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:26.984599 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:19:26.984599 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:26.984599 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:19:26.985583 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:26.985583 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:19:26.985583 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:26.985583 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:19:26.985583 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:19:26.985583 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:19:26.985583 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:19:26.985583 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:26.985583 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:26.985583 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:26.985583 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:19:26.985583 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:26.985994 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:26.985994 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:26.985994 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:19:26.985994 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:26.985994 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:19:26.985994 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:19:26.985994 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:26.985994 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:19:26.985994 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:26.985994 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:19:26.985994 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:19:26.985994 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:26.985994 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:19:26.986451 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:26.986451 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:19:26.986852 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:26.986852 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:19:26.986852 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:19:26.986852 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:19:26.986852 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:19:26.986852 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:19:26.986852 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:26.986852 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:19:26.986852 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:19:26.986852 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:26.986852 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:19:26.986852 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:19:26.986852 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:26.986852 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:26.986852 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:19:26.986852 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:26.987399 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:19:26.987399 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:19:26.987399 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:26.987399 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:26.987399 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:26.987399 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:19:26.987399 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:26.987399 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:26.987399 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:26.987399 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:19:26.987399 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:26.987399 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:19:26.987399 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:19:26.987399 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:26.987399 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:19:26.987399 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:26.987399 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:19:26.987399 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:19:26.987399 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:26.988016 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:19:26.988016 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:26.988016 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:19:31.399305 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:19:31.401178 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:19:31.401386 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:19:31.403646 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:19:31.442982 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:19:31.443252 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:31.443915 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:19:31.443915 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:31.443915 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:19:31.443915 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:19:31.443915 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:31.443915 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:19:31.443915 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:31.443915 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:19:31.444782 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_fdb_aging_settings failed after 0.043 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:31.444782 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:19:31.444782 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:31.444782 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:19:31.444782 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:19:31.444782 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:19:31.444782 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:19:31.444782 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:31.444782 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:31.444782 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:31.444782 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:19:31.444782 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:31.445129 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:31.445129 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:31.445129 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:19:31.445129 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:31.445129 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:19:31.445129 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:19:31.445129 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:31.445129 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:19:31.445129 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:31.445129 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:19:31.445129 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:19:31.445129 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:31.445129 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:19:31.445469 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:31.445469 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:19:31.445848 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:31.445848 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:19:31.445848 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:19:31.445848 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:19:31.445848 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:19:31.445848 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:19:31.445848 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:31.445848 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:19:31.445848 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:19:31.445848 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:31.445848 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:19:31.445848 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:19:31.445848 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:31.445848 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:31.445848 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:19:31.445848 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:31.446334 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:19:31.446334 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:19:31.446334 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:31.446334 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:31.446334 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:31.446334 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:19:31.446334 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:31.446334 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:31.446334 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:31.446334 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:19:31.446334 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:31.446334 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:19:31.446334 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:19:31.446334 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:31.446334 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:19:31.446334 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:31.446334 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:19:31.446334 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:19:31.446334 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:31.447038 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:19:31.447038 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:31.447038 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:19:31.943904 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:19:31.944419 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:19:31.944608 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:19:31.945547 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b027d63e-13a5-445b-84a0-8ae6223e05d5, routes=[], if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:19:31.945735 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:31.946425 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:19:31.946425 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:31.946425 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:19:31.946425 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:19:31.946425 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:31.946425 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:19:31.946425 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:31.946425 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:19:31.948671 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:31.948671 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:19:31.948671 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:31.948671 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:19:31.948671 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:19:31.948671 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:19:31.948671 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:19:31.948671 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:31.948671 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:31.948671 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:31.948671 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:19:31.948671 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:31.949402 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:31.949402 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:31.949402 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:19:31.949402 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:31.949402 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:19:31.949402 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:19:31.949402 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:31.949402 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:19:31.949402 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:31.949402 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:19:31.949402 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:19:31.949402 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:31.949402 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:19:31.950006 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:31.950006 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:19:31.951358 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:31.951358 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:19:31.951358 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:19:31.951358 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:19:31.951358 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:19:31.951358 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:19:31.951358 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:31.951358 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:19:31.951358 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:19:31.951358 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:31.951358 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:19:31.951358 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:19:31.951358 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:31.951358 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:31.951358 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:19:31.951358 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:31.951939 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:19:31.951939 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:19:31.951939 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:31.951939 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:31.951939 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:31.951939 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:19:31.951939 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:31.951939 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:31.951939 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:31.951939 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:19:31.951939 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:31.951939 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:19:31.951939 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:19:31.951939 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:31.951939 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:19:31.951939 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:31.951939 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:19:31.951939 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:19:31.951939 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:31.952491 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:19:31.952491 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:31.952491 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:19:31.958413 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:19:31.958413 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:19:31.958413 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:19:31.958413 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105458) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 15 10:19:31.959252 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:19:31.959543 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:31.960345 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:19:31.960345 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:31.960345 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:19:31.960345 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:19:31.960345 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:31.960345 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:19:31.960345 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:31.960345 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:19:31.961416 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:31.961416 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:19:31.961416 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:19:31.961416 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:19:31.961416 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:31.961416 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:19:31.961416 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:31.961416 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:19:31.961416 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:31.961416 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:19:31.961416 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:31.961416 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:19:31.961416 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:31.961416 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:19:31.961416 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:19:31.961837 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:31.961837 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:19:31.961837 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:31.961837 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:19:31.961837 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:19:31.961837 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:31.961837 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:19:31.961837 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:31.961837 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:19:31.962576 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:31.962576 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:19:31.962576 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:31.962576 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:19:31.962576 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:19:31.962576 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:19:31.962576 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:19:31.962576 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 15 10:19:31.962576 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:19:31.962576 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:19:31.962576 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:31.962576 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:31.963133 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:31.963133 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:19:31.963133 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:31.963133 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:31.963133 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:31.963133 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:19:31.963133 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:31.963133 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:19:31.963133 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:19:31.963133 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:31.963133 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:19:31.963133 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:31.963133 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:19:31.963133 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:19:31.963511 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:31.963511 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:19:31.963511 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:31.963511 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:19:31.964207 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:31.964207 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:19:31.964207 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:19:31.964207 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:19:31.964207 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:19:31.964207 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:19:31.964207 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:31.964207 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:19:31.964207 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:19:31.964207 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:31.964207 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:19:31.964207 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:19:31.964207 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:31.964207 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:31.964207 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:19:31.964207 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:31.964639 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:19:31.964639 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:19:31.964639 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^ May 15 10:19:31.964639 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:19:31.964639 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:19:31.964639 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:31.964639 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:31.964639 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:31.964639 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:19:31.964639 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:31.964639 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:31.964639 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:31.964639 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:19:31.964639 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:31.964639 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:19:31.964639 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:19:31.964639 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:31.964639 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:19:31.965204 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:31.965204 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:19:31.965204 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:19:31.965204 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:31.965204 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:19:31.965204 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:31.965204 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:19:31.970375 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:19:31.970724 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_ha_failover {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:19:31.970829 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:19:31.970943 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:19:31.971356 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:19:31.971669 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:31.972392 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:19:31.972392 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:31.972392 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:19:31.972392 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:19:31.972392 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:31.972392 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:19:31.972392 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:31.972392 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:19:31.973119 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:31.973119 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:19:31.973119 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:19:31.973119 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:19:31.973119 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:31.973119 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:19:31.973119 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:31.973119 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:19:31.973119 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:31.973119 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:19:31.973119 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:31.973119 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:19:31.973119 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:31.973119 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:19:31.973119 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:19:31.973862 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:31.973862 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:19:31.973862 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:31.973862 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:19:31.973862 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:19:31.973862 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:31.973862 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:19:31.973862 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:31.973862 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:19:31.974255 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:31.974255 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:19:31.974255 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:31.974255 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:19:31.974255 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:19:31.974255 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:19:31.974255 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 15 10:19:31.974255 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:19:31.974255 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:19:31.974255 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:31.974255 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:31.974870 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:31.974870 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:19:31.974870 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:31.974870 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:31.974870 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:31.974870 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:19:31.974870 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:31.974870 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:19:31.974870 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:19:31.974870 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:31.974870 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:19:31.974870 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:31.974870 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:19:31.974870 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:19:31.975581 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:31.975581 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:19:31.975581 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:31.975581 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:19:31.975581 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:31.975581 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:19:31.975581 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:19:31.975581 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:19:31.975581 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:19:31.975581 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:19:31.975581 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:31.975581 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:19:31.975581 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:19:31.975581 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:31.976264 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:19:31.976264 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:19:31.976264 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:31.976264 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:31.976264 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:19:31.976264 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:31.976264 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:19:31.976264 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 15 10:19:31.976264 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:19:31.976264 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:19:31.976264 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:31.976264 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:31.976264 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:31.976264 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:19:31.976264 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:31.976264 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:31.976264 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:31.976264 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:19:31.976264 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:31.977341 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:19:31.977341 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:19:31.977341 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:31.977341 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:19:31.977341 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:31.977341 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:19:31.977341 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:19:31.977341 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:31.977341 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:19:31.977341 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:31.977341 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:19:31.982160 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:19:31.982463 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:19:31.982569 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:19:31.982805 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:19:31.983141 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:19:31.983500 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:19:31.983671 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:31.984277 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:19:31.984277 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:31.984277 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:19:31.984277 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:19:31.984277 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:31.984277 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:19:31.984277 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:31.984277 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:19:31.985099 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:31.985099 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:19:31.985099 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:31.985099 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:19:31.985099 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:19:31.985099 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:19:31.985099 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:19:31.985099 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:31.985099 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:31.985099 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:31.985099 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:19:31.985099 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:31.985592 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:31.985592 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:31.985592 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:19:31.985592 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:31.985592 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:19:31.985592 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:19:31.985592 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:31.985592 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:19:31.985592 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:31.985592 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:19:31.985592 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:19:31.985592 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:31.985592 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:19:31.986122 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:31.986122 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:19:31.986122 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:31.986122 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:19:31.986122 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:19:31.986122 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:19:31.986122 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:19:31.986122 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:19:31.986122 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:31.986122 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:19:31.986122 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:19:31.986122 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:31.986122 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:19:31.986122 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:19:31.986122 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:31.986752 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:31.986752 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:19:31.986752 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:31.986752 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:19:31.986752 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:19:31.986752 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:31.986752 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:31.986752 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:31.986752 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:19:31.986752 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:31.986752 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:31.986752 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:31.986752 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:19:31.986752 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:31.986752 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:19:31.986752 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:19:31.986752 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:31.986752 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:19:31.986752 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:31.987470 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:19:31.987470 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:19:31.987470 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:31.987470 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:19:31.987470 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:31.987470 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:19:36.403652 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:19:36.403652 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:19:36.403652 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:19:36.403652 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:19:36.452978 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:19:36.453295 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:36.454750 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:19:36.454750 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:36.454750 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:19:36.454750 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:19:36.454750 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:36.454750 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:19:36.454750 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:36.454750 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:19:36.455727 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_fdb_aging_settings failed after 0.053 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:36.455727 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:19:36.455727 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:36.455727 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:19:36.455727 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:19:36.455727 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:19:36.455727 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:19:36.455727 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:36.455727 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:36.455727 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:36.455727 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:19:36.455727 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:36.455999 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:36.455999 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:36.455999 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:19:36.455999 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:36.455999 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:19:36.455999 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:19:36.455999 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:36.455999 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:19:36.455999 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:36.455999 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:19:36.455999 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:19:36.455999 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:36.455999 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:19:36.456277 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:36.456277 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:19:36.456813 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:36.456813 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:19:36.456813 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:19:36.456813 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:19:36.456813 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:19:36.456813 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:19:36.456813 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:36.456813 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:19:36.456813 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:19:36.456813 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:36.456813 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:19:36.456813 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:19:36.456813 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:36.456813 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:36.456813 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:19:36.456813 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:36.457134 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:19:36.457134 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:19:36.457134 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:36.457134 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:36.457134 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:36.457134 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:19:36.457134 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:36.457134 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:36.457134 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:36.457134 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:19:36.457134 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:36.457134 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:19:36.457134 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:19:36.457134 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:36.457134 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:19:36.457134 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:36.457134 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:19:36.457134 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:19:36.457134 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:36.457482 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:19:36.457482 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:36.457482 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:19:36.944642 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:19:36.947469 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:19:36.947730 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:19:36.948484 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b027d63e-13a5-445b-84a0-8ae6223e05d5, routes=[], if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:19:36.948778 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:36.949468 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:19:36.949468 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:36.949468 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:19:36.949468 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:19:36.949468 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:36.949468 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:19:36.949468 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:36.949468 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:19:36.950587 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:36.950587 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:19:36.950587 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:36.950587 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:19:36.950587 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:19:36.950587 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:19:36.950587 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:19:36.950587 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:36.950587 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:36.950587 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:36.950587 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:19:36.950587 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:36.951156 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:36.951156 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:36.951156 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:19:36.951156 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:36.951156 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:19:36.951156 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:19:36.951156 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:36.951156 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:19:36.951156 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:36.951156 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:19:36.951156 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:19:36.951156 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:36.951156 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:19:36.951929 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:36.951929 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:19:36.951929 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:36.951929 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:19:36.951929 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:19:36.951929 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:19:36.951929 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:19:36.951929 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:19:36.951929 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:36.951929 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:19:36.951929 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:19:36.951929 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:36.951929 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:19:36.951929 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:19:36.951929 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:36.952725 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:36.952725 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:19:36.952725 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:36.952725 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:19:36.952725 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:19:36.952725 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:36.952725 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:36.952725 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:36.952725 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:19:36.952725 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:36.952725 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:36.952725 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:36.952725 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:19:36.952725 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:36.952725 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:19:36.952725 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:19:36.952725 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:36.952725 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:19:36.952725 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:36.955054 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:19:36.955054 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:19:36.955054 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:36.955054 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:19:36.955054 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:36.955054 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:19:36.958561 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:19:36.959053 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:19:36.959160 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:19:36.959294 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105458) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 15 10:19:36.960096 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:19:36.960413 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:36.961256 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:19:36.961256 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:36.961256 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:19:36.961256 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:19:36.961256 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:36.961256 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:19:36.961256 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:36.961256 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:19:36.962265 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:36.962265 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:19:36.962265 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:19:36.962265 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:19:36.962265 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:36.962265 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:19:36.962265 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:36.962265 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:19:36.962265 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:36.962265 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:19:36.962265 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:36.962265 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:19:36.962265 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:36.962265 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:19:36.962265 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:19:36.963183 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:36.963183 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:19:36.963183 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:36.963183 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:19:36.963183 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:19:36.963183 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:36.963183 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:19:36.963183 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:36.963183 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:19:36.963609 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:36.963609 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:19:36.963609 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:36.963609 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:19:36.963609 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:19:36.963609 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:19:36.963609 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:19:36.963609 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 15 10:19:36.963609 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:19:36.963609 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:19:36.963609 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:36.963609 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:36.963887 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:36.963887 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:19:36.963887 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:36.963887 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:36.963887 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:36.963887 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:19:36.963887 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:36.963887 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:19:36.963887 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:19:36.963887 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:36.963887 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:19:36.963887 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:36.963887 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:19:36.963887 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:19:36.964147 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:36.964147 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:19:36.964147 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:36.964147 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:19:36.964421 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:36.964421 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:19:36.964421 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:19:36.964421 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:19:36.964421 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:19:36.964421 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:19:36.964421 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:36.964421 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:19:36.964421 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:19:36.964421 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:36.964421 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:19:36.964421 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:19:36.964421 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:36.964421 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:36.964421 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:19:36.964421 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:36.965228 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:19:36.965228 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:19:36.965228 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^ May 15 10:19:36.965228 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:19:36.965228 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:19:36.965228 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:36.965228 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:36.965228 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:36.965228 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:19:36.965228 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:36.965228 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:36.965228 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:36.965228 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:19:36.965228 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:36.965228 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:19:36.965228 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:19:36.965228 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:36.965228 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:19:36.965833 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:36.965833 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:19:36.965833 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:19:36.965833 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:36.965833 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:19:36.965833 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:36.965833 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:19:36.971126 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:19:36.971511 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_ha_failover {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:19:36.971635 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:19:36.971843 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:19:36.972469 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:19:36.972897 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:36.973775 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:19:36.973775 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:36.973775 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:19:36.973775 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:19:36.973775 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:36.973775 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:19:36.973775 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:36.973775 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:19:36.974665 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:36.974665 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:19:36.974665 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:19:36.974665 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:19:36.974665 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:36.974665 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:19:36.974665 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:36.974665 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:19:36.974665 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:36.974665 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:19:36.974665 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:36.974665 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:19:36.974665 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:36.974665 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:19:36.974665 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:19:36.975324 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:36.975324 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:19:36.975324 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:36.975324 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:19:36.975324 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:19:36.975324 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:36.975324 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:19:36.975324 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:36.975324 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:19:36.975728 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:36.975728 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:19:36.975728 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:36.975728 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:19:36.975728 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:19:36.975728 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:19:36.975728 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 15 10:19:36.975728 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:19:36.975728 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:19:36.975728 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:36.975728 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:36.976312 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:36.976312 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:19:36.976312 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:36.976312 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:36.976312 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:36.976312 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:19:36.976312 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:36.976312 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:19:36.976312 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:19:36.976312 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:36.976312 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:19:36.976312 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:36.976312 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:19:36.976312 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:19:36.977036 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:36.977036 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:19:36.977036 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:36.977036 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:19:36.977036 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:36.977036 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:19:36.977036 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:19:36.977036 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:19:36.977036 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:19:36.977036 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:19:36.977036 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:36.977036 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:19:36.977036 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:19:36.977036 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:36.978662 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:19:36.978662 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:19:36.978662 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:36.978662 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:36.978662 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:19:36.978662 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:36.978662 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:19:36.978662 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 15 10:19:36.978662 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:19:36.978662 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:19:36.978662 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:36.978662 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:36.978662 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:36.978662 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:19:36.978662 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:36.978662 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:36.978662 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:36.978662 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:19:36.978662 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:36.980945 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:19:36.980945 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:19:36.980945 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:36.980945 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:19:36.980945 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:36.980945 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:19:36.980945 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:19:36.980945 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:36.980945 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:19:36.980945 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:36.980945 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:19:36.982892 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:19:36.983265 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:19:36.983403 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:19:36.983590 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:19:36.984245 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:19:36.984505 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:19:36.984690 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:36.985560 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:19:36.985560 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:36.985560 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:19:36.985560 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:19:36.985560 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:36.985560 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:19:36.985560 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:36.985560 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:19:36.986575 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:36.986575 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:19:36.986575 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:36.986575 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:19:36.986575 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:19:36.986575 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:19:36.986575 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:19:36.986575 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:36.986575 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:36.986575 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:36.986575 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:19:36.986575 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:36.987146 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:36.987146 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:36.987146 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:19:36.987146 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:36.987146 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:19:36.987146 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:19:36.987146 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:36.987146 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:19:36.987146 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:36.987146 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:19:36.987146 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:19:36.987146 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:36.987146 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:19:36.987555 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:36.987555 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:19:36.987710 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:36.987710 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:19:36.987710 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:19:36.987710 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:19:36.987710 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:19:36.987710 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:19:36.987710 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:36.987710 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:19:36.987710 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:19:36.987710 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:36.987710 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:19:36.987710 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:19:36.987710 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:36.987710 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:36.987710 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:19:36.987710 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:36.988434 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:19:36.988434 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:19:36.988434 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:36.988434 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:36.988434 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:36.988434 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:19:36.988434 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:36.988434 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:36.988434 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:36.988434 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:19:36.988434 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:36.988434 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:19:36.988434 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:19:36.988434 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:36.988434 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:19:36.988434 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:36.988434 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:19:36.988434 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:19:36.988434 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:36.988971 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:19:36.988971 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:36.988971 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:19:41.403310 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:19:41.403774 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:19:41.403936 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:19:41.405448 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:19:41.442138 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:19:41.442753 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:41.443351 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:19:41.443351 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:41.443351 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:19:41.443351 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:19:41.443351 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:41.443351 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:19:41.443351 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:41.443351 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:19:41.444540 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_fdb_aging_settings failed after 0.039 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:41.444540 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:19:41.444540 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:41.444540 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:19:41.444540 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:19:41.444540 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:19:41.444540 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:19:41.444540 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:41.444540 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:41.444540 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:41.444540 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:19:41.444540 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:41.445202 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:41.445202 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:41.445202 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:19:41.445202 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:41.445202 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:19:41.445202 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:19:41.445202 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:41.445202 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:19:41.445202 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:41.445202 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:19:41.445202 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:19:41.445202 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:41.445202 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:19:41.445816 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:41.445816 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:19:41.445816 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:41.445816 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:19:41.445816 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:19:41.445816 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:19:41.445816 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:19:41.445816 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:19:41.445816 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:41.445816 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:19:41.445816 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:19:41.445816 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:41.445816 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:19:41.445816 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:19:41.445816 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:41.446267 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:41.446267 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:19:41.446267 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:41.446267 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:19:41.446267 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:19:41.446267 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:41.446267 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:41.446267 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:41.446267 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:19:41.446267 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:41.446267 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:41.446267 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:41.446267 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:19:41.446267 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:41.446267 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:19:41.446267 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:19:41.446267 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:41.446267 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:19:41.446267 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:41.447090 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:19:41.447090 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:19:41.447090 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:41.447090 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:19:41.447090 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:41.447090 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:19:41.947505 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:19:41.948801 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:19:41.948801 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:19:41.949197 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b027d63e-13a5-445b-84a0-8ae6223e05d5, routes=[], if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:19:41.949441 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:41.953636 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:19:41.953636 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:41.953636 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:19:41.953636 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:19:41.953636 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:41.953636 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:19:41.953636 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:41.953636 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:19:41.953636 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:41.953636 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:19:41.953636 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:41.953636 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:19:41.953636 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:19:41.953636 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:19:41.953636 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:19:41.954536 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:41.954536 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:41.954536 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:41.954536 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:19:41.954536 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:41.954536 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:41.954536 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:41.954536 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:19:41.954536 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:41.954536 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:19:41.954536 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:19:41.954536 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:41.954536 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:19:41.954536 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:41.955188 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:19:41.955188 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:19:41.955188 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:41.955188 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:19:41.955188 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:41.955188 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:19:41.955188 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:41.955188 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:19:41.955188 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:19:41.955188 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:19:41.955188 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:19:41.955188 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:19:41.955188 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:41.955188 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:19:41.955611 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:19:41.955611 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:41.955611 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:19:41.955611 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:19:41.955611 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:41.955611 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:41.955611 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:19:41.955611 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:41.955611 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:19:41.955611 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:19:41.955611 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:41.955611 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:41.955611 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:41.955611 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:19:41.955611 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:41.955611 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:41.955611 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:41.955611 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:19:41.955611 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:41.955611 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:19:41.956223 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:19:41.956223 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:41.956223 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:19:41.956223 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:41.956223 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:19:41.956223 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:19:41.956223 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:41.956223 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:19:41.956223 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:41.956223 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:19:41.959186 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:19:41.959582 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:19:41.959830 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:19:41.960087 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105458) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 15 10:19:41.960739 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:19:41.960962 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:41.964550 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:19:41.964550 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:41.964550 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:19:41.964550 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:19:41.964550 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:41.964550 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:19:41.964550 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:41.964550 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:19:41.964550 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:41.964550 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:19:41.964550 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:19:41.964550 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:19:41.964550 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:41.964550 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:19:41.964550 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:41.964550 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:19:41.964550 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:41.964550 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:19:41.965258 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:41.965258 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:19:41.965258 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:41.965258 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:19:41.965258 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:19:41.965258 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:41.965258 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:19:41.965258 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:41.965258 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:19:41.965258 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:19:41.965258 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:41.965258 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:19:41.965258 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:41.965258 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:19:41.965258 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:41.965746 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:19:41.965746 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:41.965746 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:19:41.965746 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:19:41.965746 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:19:41.965746 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:19:41.965746 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 15 10:19:41.965746 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:19:41.965746 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:19:41.965746 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:41.965746 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:41.965746 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:41.965746 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:19:41.966164 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:41.966164 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:41.966164 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:41.966164 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:19:41.966164 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:41.966164 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:19:41.966164 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:19:41.966164 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:41.966164 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:19:41.966164 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:41.966164 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:19:41.966164 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:19:41.966164 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:41.966164 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:19:41.966582 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:41.966582 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:19:41.966582 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:41.966582 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:19:41.966582 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:19:41.966582 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:19:41.966582 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:19:41.966582 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:19:41.966582 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:41.966582 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:19:41.966582 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:19:41.966582 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:41.966582 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:19:41.966582 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:19:41.966582 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:41.967062 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:41.967062 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:19:41.967062 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:41.967062 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:19:41.967062 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:19:41.967062 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^ May 15 10:19:41.967062 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:19:41.967062 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:19:41.967062 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:41.967062 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:41.967062 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:41.967062 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:19:41.967062 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:41.967062 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:41.967062 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:41.967062 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:19:41.967062 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:41.967062 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:19:41.967062 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:19:41.967604 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:41.967604 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:19:41.967604 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:41.967604 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:19:41.967604 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:19:41.967604 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:41.967604 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:19:41.967604 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:41.967604 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:19:41.971673 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:19:41.972413 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_ha_failover {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:19:41.972413 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:19:41.972413 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:19:41.972948 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:19:41.973240 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:41.976706 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:19:41.976706 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:41.976706 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:19:41.976706 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:19:41.976706 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:41.976706 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:19:41.976706 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:41.976706 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:19:41.976706 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:41.976706 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:19:41.976706 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:19:41.976706 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:19:41.976706 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:41.976706 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:19:41.976706 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:41.976706 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:19:41.976706 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:41.976706 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:19:41.977290 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:41.977290 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:19:41.977290 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:41.977290 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:19:41.977290 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:19:41.977290 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:41.977290 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:19:41.977290 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:41.977290 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:19:41.977290 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:19:41.977290 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:41.977290 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:19:41.977290 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:41.977290 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:19:41.977290 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:41.977780 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:19:41.977780 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:41.977780 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:19:41.977780 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:19:41.977780 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:19:41.977780 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 15 10:19:41.977780 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:19:41.977780 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:19:41.977780 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:41.977780 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:41.977780 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:41.977780 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:19:41.977780 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:41.978195 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:41.978195 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:41.978195 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:19:41.978195 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:41.978195 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:19:41.978195 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:19:41.978195 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:41.978195 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:19:41.978195 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:41.978195 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:19:41.978195 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:19:41.978195 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:41.978195 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:19:41.978578 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:41.978578 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:19:41.979016 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:41.979016 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:19:41.979016 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:19:41.979016 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:19:41.979016 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:19:41.979016 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:19:41.979016 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:41.979016 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:19:41.979016 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:19:41.979016 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:41.979016 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:19:41.979016 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:19:41.979016 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:41.979016 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:41.979016 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:19:41.979016 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:41.979016 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:19:41.979581 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 15 10:19:41.979581 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:19:41.979581 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:19:41.979581 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:41.979581 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:41.979581 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:41.979581 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:19:41.979581 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:41.979581 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:41.979581 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:41.979581 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:19:41.979581 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:41.979581 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:19:41.979581 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:19:41.979581 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:41.979581 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:19:41.979581 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:41.979581 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:19:41.979581 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:19:41.980140 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:41.980140 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:19:41.980140 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:41.980140 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:19:41.983405 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:19:41.984144 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:19:41.984144 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:19:41.984144 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:19:41.984711 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:19:41.985121 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:19:41.985303 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:41.988156 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:19:41.988156 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:41.988156 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:19:41.988156 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:19:41.988156 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:41.988156 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:19:41.988156 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:41.988156 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:19:41.988156 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:41.988156 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:19:41.988156 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:41.988156 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:19:41.988156 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:19:41.988156 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:19:41.988156 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:19:41.988656 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:41.988656 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:41.988656 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:41.988656 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:19:41.988656 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:41.988656 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:41.988656 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:41.988656 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:19:41.988656 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:41.988656 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:19:41.988656 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:19:41.988656 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:41.988656 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:19:41.988656 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:41.989163 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:19:41.989163 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:19:41.989163 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:41.989163 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:19:41.989163 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:41.989163 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:19:41.989163 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:41.989163 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:19:41.989163 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:19:41.989163 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:19:41.989163 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:19:41.989163 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:19:41.989163 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:41.989163 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:19:41.989582 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:19:41.989582 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:41.989582 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:19:41.989582 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:19:41.989582 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:41.989582 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:41.989582 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:19:41.989582 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:41.989582 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:19:41.989582 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:19:41.989582 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:41.989582 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:41.989582 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:41.989582 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:19:41.989582 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:41.989582 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:41.989582 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:41.989582 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:19:41.989582 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:41.989582 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:19:41.990284 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:19:41.990284 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:41.990284 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:19:41.990284 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:41.990284 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:19:41.990284 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:19:41.990284 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:41.990284 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:19:41.990284 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:41.990284 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:19:46.404956 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:19:46.404956 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:19:46.404956 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:19:46.407585 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:19:46.408612 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG dbcounter [-] [105458] Writing DB stats neutron:SELECT=580,neutron:INSERT=1,neutron:UPDATE=2,neutron:DELETE=1 {{(pid=105458) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} May 15 10:19:46.451873 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:19:46.452308 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:46.453375 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:19:46.453375 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:46.453375 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:19:46.453375 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:19:46.453375 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:46.453375 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:19:46.453375 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:46.453375 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:19:46.454714 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_fdb_aging_settings failed after 0.048 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:46.454714 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:19:46.454714 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:46.454714 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:19:46.454714 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:19:46.454714 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:19:46.454714 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:19:46.454714 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:46.454714 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:46.454714 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:46.454714 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:19:46.454714 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:46.455233 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:46.455233 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:46.455233 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:19:46.455233 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:46.455233 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:19:46.455233 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:19:46.455233 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:46.455233 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:19:46.455233 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:46.455233 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:19:46.455233 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:19:46.455233 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:46.455233 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:19:46.455990 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:46.455990 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:19:46.456129 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:46.456129 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:19:46.456129 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:19:46.456129 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:19:46.456129 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:19:46.456129 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:19:46.456129 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:46.456129 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:19:46.456129 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:19:46.456129 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:46.456129 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:19:46.456129 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:19:46.456129 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:46.456129 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:46.456129 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:19:46.456129 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:46.456877 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:19:46.456877 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:19:46.456877 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:46.456877 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:46.456877 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:46.456877 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:19:46.456877 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:46.456877 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:46.456877 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:46.456877 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:19:46.456877 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:46.456877 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:19:46.456877 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:19:46.456877 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:46.456877 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:19:46.456877 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:46.456877 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:19:46.456877 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:19:46.456877 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:46.457632 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:19:46.457632 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:46.457632 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:19:46.948722 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:19:46.949299 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:19:46.949555 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:19:46.950530 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b027d63e-13a5-445b-84a0-8ae6223e05d5, routes=[], if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:19:46.950962 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:46.951929 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:19:46.951929 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:46.951929 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:19:46.951929 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:19:46.951929 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:46.951929 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:19:46.951929 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:46.951929 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:19:46.953661 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:46.953661 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:19:46.953661 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:46.953661 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:19:46.953661 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:19:46.953661 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:19:46.953661 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:19:46.953661 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:46.953661 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:46.953661 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:46.953661 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:19:46.953661 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:46.954398 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:46.954398 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:46.954398 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:19:46.954398 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:46.954398 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:19:46.954398 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:19:46.954398 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:46.954398 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:19:46.954398 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:46.954398 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:19:46.954398 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:19:46.954398 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:46.954398 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:19:46.955048 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:46.955048 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:19:46.955351 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:46.955351 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:19:46.955351 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:19:46.955351 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:19:46.955351 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:19:46.955351 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:19:46.955351 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:46.955351 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:19:46.955351 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:19:46.955351 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:46.955351 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:19:46.955351 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:19:46.955351 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:46.955351 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:46.955351 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:19:46.955351 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:46.956091 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:19:46.956091 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:19:46.956091 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:46.956091 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:46.956091 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:46.956091 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:19:46.956091 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:46.956091 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:46.956091 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:46.956091 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:19:46.956091 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:46.956091 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:19:46.956091 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:19:46.956091 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:46.956091 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:19:46.956091 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:46.956091 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:19:46.956091 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:19:46.956091 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:46.956998 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:19:46.956998 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:46.956998 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:19:46.959862 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:19:46.960091 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:19:46.960255 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:19:46.960376 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105458) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 15 10:19:46.960747 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:19:46.960942 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:46.961446 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:19:46.961446 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:46.961446 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:19:46.961446 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:19:46.961446 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:46.961446 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:19:46.961446 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:46.961446 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:19:46.962287 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:46.962287 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:19:46.962287 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:19:46.962287 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:19:46.962287 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:46.962287 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:19:46.962287 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:46.962287 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:19:46.962287 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:46.962287 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:19:46.962287 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:46.962287 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:19:46.962287 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:46.962287 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:19:46.962287 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:19:46.962736 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:46.962736 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:19:46.962736 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:46.962736 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:19:46.962736 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:19:46.962736 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:46.962736 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:19:46.962736 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:46.962736 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:19:46.963176 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:46.963176 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:19:46.963176 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:46.963176 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:19:46.963176 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:19:46.963176 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:19:46.963176 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:19:46.963176 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 15 10:19:46.963176 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:19:46.963176 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:19:46.963176 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:46.963176 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:46.963565 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:46.963565 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:19:46.963565 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:46.963565 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:46.963565 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:46.963565 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:19:46.963565 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:46.963565 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:19:46.963565 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:19:46.963565 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:46.963565 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:19:46.963565 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:46.963565 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:19:46.963565 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:19:46.963956 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:46.963956 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:19:46.963956 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:46.963956 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:19:46.964217 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:46.964217 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:19:46.964217 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:19:46.964217 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:19:46.964217 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:19:46.964217 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:19:46.964217 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:46.964217 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:19:46.964217 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:19:46.964217 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:46.964217 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:19:46.964217 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:19:46.964217 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:46.964217 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:46.964217 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:19:46.964217 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:46.964716 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:19:46.964716 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:19:46.964716 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^ May 15 10:19:46.964716 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:19:46.964716 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:19:46.964716 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:46.964716 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:46.964716 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:46.964716 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:19:46.964716 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:46.964716 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:46.964716 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:46.964716 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:19:46.964716 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:46.964716 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:19:46.964716 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:19:46.964716 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:46.964716 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:19:46.965146 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:46.965146 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:19:46.965146 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:19:46.965146 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:46.965146 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:19:46.965146 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:46.965146 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:19:46.972637 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:19:46.972983 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_ha_failover {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:19:46.973106 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:19:46.973232 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:19:46.973803 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:19:46.974361 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:46.975206 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:19:46.975206 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:46.975206 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:19:46.975206 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:19:46.975206 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:46.975206 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:19:46.975206 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:46.975206 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:19:46.976764 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:46.976764 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:19:46.976764 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:19:46.976764 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:19:46.976764 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:46.976764 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:19:46.976764 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:46.976764 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:19:46.976764 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:46.976764 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:19:46.976764 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:46.976764 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:19:46.976764 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:46.976764 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:19:46.976764 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:19:46.977709 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:46.977709 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:19:46.977709 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:46.977709 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:19:46.977709 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:19:46.977709 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:46.977709 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:19:46.977709 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:46.977709 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:19:46.978858 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_ha_failover failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:46.978858 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:19:46.978858 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:46.978858 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:19:46.978858 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:19:46.978858 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:19:46.978858 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 15 10:19:46.978858 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:19:46.978858 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:19:46.978858 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:46.978858 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:46.979460 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:46.979460 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:19:46.979460 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:46.979460 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:46.979460 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:46.979460 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:19:46.979460 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:46.979460 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:19:46.979460 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:19:46.979460 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:46.979460 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:19:46.979460 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:46.979460 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:19:46.979460 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:19:46.980139 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:46.980139 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:19:46.980139 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:46.980139 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:19:46.981302 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:46.981302 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:19:46.981302 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:19:46.981302 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:19:46.981302 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:19:46.981302 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:19:46.981302 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:46.981302 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:19:46.981302 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:19:46.981302 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:46.981302 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:19:46.981302 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:19:46.981302 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:46.981302 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:46.981302 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:19:46.981302 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:46.981302 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:19:46.982071 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 15 10:19:46.982071 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:19:46.982071 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:19:46.982071 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:46.982071 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:46.982071 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:46.982071 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:19:46.982071 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:46.982071 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:46.982071 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:46.982071 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:19:46.982071 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:46.982071 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:19:46.982071 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:19:46.982071 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:46.982071 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:19:46.982071 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:46.982071 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:19:46.982071 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:19:46.982816 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:46.982816 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:19:46.982816 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:46.982816 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:19:46.984045 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:19:46.984389 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:19:46.984554 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:19:46.984796 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:19:46.985318 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:19:46.985723 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:19:46.985953 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:46.987025 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:19:46.987025 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:46.987025 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:19:46.987025 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:19:46.987025 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:46.987025 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:19:46.987025 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:46.987025 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:19:46.989074 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:46.989074 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:19:46.989074 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:46.989074 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:19:46.989074 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:19:46.989074 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:19:46.989074 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:19:46.989074 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:46.989074 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:46.989074 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:46.989074 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:19:46.989074 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:46.989525 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:46.989525 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:46.989525 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:19:46.989525 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:46.989525 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:19:46.989525 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:19:46.989525 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:46.989525 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:19:46.989525 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:46.989525 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:19:46.989525 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:19:46.989525 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:46.989525 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:19:46.989951 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:46.989951 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:19:46.990721 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:46.990721 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:19:46.990721 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:19:46.990721 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:19:46.990721 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:19:46.990721 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:19:46.990721 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:46.990721 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:19:46.990721 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:19:46.990721 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:46.990721 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:19:46.990721 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:19:46.990721 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:46.990721 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:46.990721 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:19:46.990721 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:46.991304 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:19:46.991304 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:19:46.991304 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:46.991304 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:46.991304 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:46.991304 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:19:46.991304 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:46.991304 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:46.991304 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:46.991304 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:19:46.991304 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:46.991304 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:19:46.991304 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:19:46.991304 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:46.991304 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:19:46.991304 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:46.991304 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:19:46.991304 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:19:46.991304 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:46.991864 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:19:46.991864 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:46.991864 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:19:51.405207 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:19:51.405950 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:19:51.405950 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:19:51.406778 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:19:51.445075 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:19:51.445235 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:51.447794 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:19:51.447794 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:51.447794 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:19:51.447794 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:19:51.447794 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:51.447794 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:19:51.447794 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:51.447794 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:19:51.447794 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_fdb_aging_settings failed after 0.040 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:51.447794 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:19:51.447794 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:51.447794 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:19:51.447794 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:19:51.447794 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:19:51.447794 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:19:51.448918 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:51.448918 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:51.448918 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:51.448918 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:19:51.448918 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:51.448918 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:51.448918 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:51.448918 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:19:51.448918 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:51.448918 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:19:51.448918 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:19:51.448918 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:51.448918 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:19:51.448918 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:51.449572 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:19:51.449572 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:19:51.449572 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:51.449572 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:19:51.449572 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:51.449572 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:19:51.449572 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:51.449572 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:19:51.449572 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:19:51.449572 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:19:51.449572 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:19:51.449572 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:19:51.449572 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:51.449572 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:19:51.450393 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:19:51.450393 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:51.450393 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:19:51.450393 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:19:51.450393 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:51.450393 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:51.450393 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:19:51.450393 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:51.450393 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:19:51.450393 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:19:51.450393 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:51.450393 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:51.450393 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:51.450393 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:19:51.450393 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:51.450393 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:51.450393 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:51.450393 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:19:51.450393 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:51.450393 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:19:51.450393 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:19:51.451497 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:51.451497 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:19:51.451497 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:51.451497 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:19:51.451497 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:19:51.451497 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:51.451497 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:19:51.451497 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:51.451497 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:19:51.949746 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:19:51.950229 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:19:51.950229 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:19:51.951028 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b027d63e-13a5-445b-84a0-8ae6223e05d5, routes=[], if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:19:51.951419 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:51.952413 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:19:51.952413 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:51.952413 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:19:51.952413 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:19:51.952413 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:51.952413 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:19:51.952413 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:51.952413 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:19:51.953027 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:51.953027 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:19:51.953027 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:51.953027 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:19:51.953027 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:19:51.953027 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:19:51.953027 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:19:51.953027 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:51.953027 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:51.953027 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:51.953027 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:19:51.953027 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:51.953571 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:51.953571 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:51.953571 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:19:51.953571 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:51.953571 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:19:51.953571 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:19:51.953571 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:51.953571 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:19:51.953571 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:51.953571 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:19:51.953571 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:19:51.953571 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:51.953571 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:19:51.954173 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:51.954173 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:19:51.954173 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:51.954173 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:19:51.954173 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:19:51.954173 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:19:51.954173 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:19:51.954173 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:19:51.954173 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:51.954173 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:19:51.954173 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:19:51.954173 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:51.954173 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:19:51.954173 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:19:51.954173 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:51.954858 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:51.954858 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:19:51.954858 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:51.954858 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:19:51.954858 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:19:51.954858 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:51.954858 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:51.954858 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:51.954858 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:19:51.954858 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:51.954858 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:51.954858 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:51.954858 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:19:51.954858 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:51.954858 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:19:51.954858 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:19:51.954858 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:51.954858 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:19:51.954858 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:51.956010 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:19:51.956010 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:19:51.956010 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:51.956010 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:19:51.956010 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:51.956010 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:19:51.960510 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:19:51.960874 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:19:51.960947 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:19:51.961032 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105458) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 15 10:19:51.962031 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:19:51.962278 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:51.962821 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:19:51.962821 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:51.962821 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:19:51.962821 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:19:51.962821 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:51.962821 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:19:51.962821 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:51.962821 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:19:51.963327 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:51.963327 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:19:51.963327 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:19:51.963327 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:19:51.963327 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:51.963327 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:19:51.963327 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:51.963327 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:19:51.963327 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:51.963327 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:19:51.963327 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:51.963327 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:19:51.963327 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:51.963327 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:19:51.963327 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:19:51.964066 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:51.964066 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:19:51.964066 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:51.964066 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:19:51.964066 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:19:51.964066 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:51.964066 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:19:51.964066 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:51.964066 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:19:51.964066 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:51.964066 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:19:51.964066 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:51.964066 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:19:51.964066 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:19:51.964769 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:19:51.964769 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:19:51.964769 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 15 10:19:51.964769 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:19:51.964769 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:19:51.964769 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:51.964769 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:51.964769 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:51.964769 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:19:51.964769 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:51.964769 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:51.964769 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:51.964769 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:19:51.965366 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:51.965366 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:19:51.965366 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:19:51.965366 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:51.965366 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:19:51.965366 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:51.965366 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:19:51.965366 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:19:51.965366 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:51.965366 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:19:51.965366 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:51.965366 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:19:51.965928 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:51.965928 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:19:51.965928 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:19:51.965928 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:19:51.965928 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:19:51.965928 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:19:51.965928 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:51.965928 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:19:51.965928 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:19:51.965928 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:51.965928 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:19:51.965928 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:19:51.965928 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:51.965928 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:51.965928 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:19:51.965928 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:51.966603 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:19:51.966603 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:19:51.966603 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^ May 15 10:19:51.966603 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:19:51.966603 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:19:51.966603 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:51.966603 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:51.966603 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:51.966603 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:19:51.966603 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:51.966603 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:51.966603 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:51.966603 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:19:51.966603 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:51.966603 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:19:51.966603 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:19:51.966603 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:51.966603 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:19:51.967167 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:51.967167 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:19:51.967167 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:19:51.967167 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:51.967167 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:19:51.967167 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:51.967167 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:19:51.973393 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:19:51.973855 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_ha_failover {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:19:51.974048 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:19:51.974254 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:19:51.974651 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:19:51.974887 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:51.975343 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:19:51.975343 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:51.975343 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:19:51.975343 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:19:51.975343 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:51.975343 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:19:51.975343 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:51.975343 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:19:51.976407 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:51.976407 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:19:51.976407 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:19:51.976407 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:19:51.976407 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:51.976407 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:19:51.976407 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:51.976407 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:19:51.976407 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:51.976407 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:19:51.976407 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:51.976407 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:19:51.976407 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:51.976407 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:19:51.976407 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:19:51.976727 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:51.976727 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:19:51.976727 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:51.976727 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:19:51.976727 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:19:51.976727 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:51.976727 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:19:51.976727 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:51.976727 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:19:51.977378 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:51.977378 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:19:51.977378 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:51.977378 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:19:51.977378 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:19:51.977378 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:19:51.977378 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 15 10:19:51.977378 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:19:51.977378 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:19:51.977378 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:51.977378 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:51.977594 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:51.977594 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:19:51.977594 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:51.977594 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:51.977594 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:51.977594 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:19:51.977594 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:51.977594 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:19:51.977594 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:19:51.977594 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:51.977594 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:19:51.977594 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:51.977594 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:19:51.977594 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:19:51.977867 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:51.977867 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:19:51.977867 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:51.977867 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:19:51.978760 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:51.978760 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:19:51.978760 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:19:51.978760 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:19:51.978760 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:19:51.978760 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:19:51.978760 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:51.978760 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:19:51.978760 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:19:51.978760 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:51.978760 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:19:51.978760 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:19:51.978760 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:51.978760 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:51.978760 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:19:51.978760 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:51.978760 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:19:51.979249 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 15 10:19:51.979249 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:19:51.979249 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:19:51.979249 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:51.979249 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:51.979249 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:51.979249 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:19:51.979249 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:51.979249 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:51.979249 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:51.979249 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:19:51.979249 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:51.979249 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:19:51.979249 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:19:51.979249 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:51.979249 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:19:51.979249 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:51.979249 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:19:51.979249 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:19:51.980342 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:51.980342 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:19:51.980342 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:51.980342 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:19:51.984490 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:19:51.984789 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:19:51.984913 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:19:51.985074 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:19:51.985543 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:19:51.985891 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:19:51.986056 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:51.986696 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:19:51.986696 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:51.986696 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:19:51.986696 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:19:51.986696 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:51.986696 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:19:51.986696 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:51.986696 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:19:51.987581 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:51.987581 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:19:51.987581 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:51.987581 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:19:51.987581 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:19:51.987581 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:19:51.987581 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:19:51.987581 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:51.987581 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:51.987581 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:51.987581 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:19:51.987581 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:51.988014 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:51.988014 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:51.988014 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:19:51.988014 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:51.988014 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:19:51.988014 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:19:51.988014 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:51.988014 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:19:51.988014 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:51.988014 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:19:51.988014 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:19:51.988014 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:51.988014 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:19:51.988451 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:51.988451 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:19:51.988873 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:51.988873 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:19:51.988873 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:19:51.988873 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:19:51.988873 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:19:51.988873 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:19:51.988873 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:51.988873 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:19:51.988873 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:19:51.988873 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:51.988873 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:19:51.988873 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:19:51.988873 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:51.988873 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:51.988873 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:19:51.988873 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:51.989483 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:19:51.989483 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:19:51.989483 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:51.989483 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:51.989483 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:51.989483 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:19:51.989483 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:51.989483 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:51.989483 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:51.989483 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:19:51.989483 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:51.989483 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:19:51.989483 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:19:51.989483 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:51.989483 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:19:51.989483 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:51.989483 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:19:51.989483 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:19:51.989483 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:51.990370 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:19:51.990370 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:51.990370 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:19:56.405945 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:19:56.406790 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:19:56.406790 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:19:56.407367 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:19:56.442803 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:19:56.443098 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:56.443691 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:19:56.443691 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:56.443691 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:19:56.443691 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:19:56.443691 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:56.443691 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:19:56.443691 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:56.443691 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:19:56.444760 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_fdb_aging_settings failed after 0.037 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:56.444760 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:19:56.444760 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:56.444760 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:19:56.444760 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:19:56.444760 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:19:56.444760 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:19:56.444760 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:56.444760 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:56.444760 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:56.444760 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:19:56.444760 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:56.445346 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:56.445346 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:56.445346 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:19:56.445346 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:56.445346 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:19:56.445346 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:19:56.445346 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:56.445346 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:19:56.445346 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:56.445346 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:19:56.445346 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:19:56.445346 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:56.445346 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:19:56.445912 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:56.445912 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:19:56.445912 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:56.445912 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:19:56.445912 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:19:56.445912 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:19:56.445912 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:19:56.445912 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:19:56.445912 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:56.445912 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:19:56.445912 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:19:56.445912 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:56.445912 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:19:56.445912 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:19:56.445912 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:56.446594 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:56.446594 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:19:56.446594 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:56.446594 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:19:56.446594 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:19:56.446594 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:56.446594 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:56.446594 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:56.446594 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:19:56.446594 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:56.446594 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:56.446594 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:56.446594 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:19:56.446594 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:56.446594 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:19:56.446594 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:19:56.446594 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:56.446594 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:19:56.446594 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:56.447413 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:19:56.447413 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:19:56.447413 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:56.447413 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:19:56.447413 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:56.447413 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:19:56.950407 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:19:56.950783 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:19:56.950898 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:19:56.951808 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b027d63e-13a5-445b-84a0-8ae6223e05d5, routes=[], if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:19:56.952116 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:56.952938 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:19:56.952938 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:56.952938 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:19:56.952938 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:19:56.952938 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:56.952938 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:19:56.952938 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:56.952938 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:19:56.954657 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:56.954657 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:19:56.954657 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:56.954657 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:19:56.954657 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:19:56.954657 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:19:56.954657 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:19:56.954657 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:56.954657 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:56.954657 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:56.954657 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:19:56.954657 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:56.955069 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:56.955069 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:56.955069 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:19:56.955069 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:56.955069 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:19:56.955069 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:19:56.955069 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:56.955069 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:19:56.955069 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:56.955069 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:19:56.955069 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:19:56.955069 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:56.955069 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:19:56.955460 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:56.955460 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:19:56.956666 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:56.956666 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:19:56.956666 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:19:56.956666 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:19:56.956666 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:19:56.956666 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:19:56.956666 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:56.956666 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:19:56.956666 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:19:56.956666 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:56.956666 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:19:56.956666 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:19:56.956666 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:56.956666 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:56.956666 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:19:56.956666 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:56.957169 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:19:56.957169 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:19:56.957169 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:56.957169 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:56.957169 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:56.957169 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:19:56.957169 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:56.957169 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:56.957169 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:56.957169 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:19:56.957169 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:56.957169 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:19:56.957169 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:19:56.957169 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:56.957169 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:19:56.957169 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:56.957169 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:19:56.957169 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:19:56.957169 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:56.957702 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:19:56.957702 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:56.957702 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:19:56.961130 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:19:56.961569 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:19:56.961779 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:19:56.961941 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105458) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 15 10:19:56.962341 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:19:56.962548 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:56.963050 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:19:56.963050 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:56.963050 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:19:56.963050 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:19:56.963050 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:56.963050 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:19:56.963050 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:56.963050 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:19:56.963910 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:56.963910 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:19:56.963910 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:19:56.963910 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:19:56.963910 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:56.963910 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:19:56.963910 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:56.963910 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:19:56.963910 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:56.963910 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:19:56.963910 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:56.963910 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:19:56.963910 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:56.963910 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:19:56.963910 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:19:56.964367 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:56.964367 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:19:56.964367 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:56.964367 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:19:56.964367 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:19:56.964367 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:56.964367 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:19:56.964367 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:56.964367 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:19:56.965021 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:56.965021 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:19:56.965021 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:56.965021 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:19:56.965021 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:19:56.965021 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:19:56.965021 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:19:56.965021 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 15 10:19:56.965021 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:19:56.965021 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:19:56.965021 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:56.965021 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:56.965413 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:56.965413 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:19:56.965413 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:56.965413 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:56.965413 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:56.965413 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:19:56.965413 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:56.965413 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:19:56.965413 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:19:56.965413 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:56.965413 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:19:56.965413 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:56.965413 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:19:56.965413 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:19:56.965781 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:56.965781 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:19:56.965781 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:56.965781 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:19:56.966329 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:56.966329 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:19:56.966329 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:19:56.966329 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:19:56.966329 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:19:56.966329 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:19:56.966329 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:56.966329 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:19:56.966329 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:19:56.966329 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:56.966329 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:19:56.966329 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:19:56.966329 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:56.966329 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:56.966329 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:19:56.966329 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:56.966804 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:19:56.966804 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:19:56.966804 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^ May 15 10:19:56.966804 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:19:56.966804 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:19:56.966804 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:56.966804 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:56.966804 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:56.966804 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:19:56.966804 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:56.966804 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:56.966804 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:56.966804 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:19:56.966804 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:56.966804 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:19:56.966804 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:19:56.966804 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:56.966804 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:19:56.967647 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:56.967647 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:19:56.967647 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:19:56.967647 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:56.967647 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:19:56.967647 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:56.967647 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:19:56.974327 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:19:56.974835 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_ha_failover {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:19:56.974971 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:19:56.975159 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:19:56.975611 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:19:56.975884 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:56.976384 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:19:56.976384 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:56.976384 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:19:56.976384 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:19:56.976384 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:56.976384 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:19:56.976384 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:56.976384 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:19:56.977208 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:56.977208 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:19:56.977208 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:19:56.977208 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:19:56.977208 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:56.977208 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:19:56.977208 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:56.977208 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:19:56.977208 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:56.977208 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:19:56.977208 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:56.977208 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:19:56.977208 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:56.977208 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:19:56.977208 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:19:56.977653 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:56.977653 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:19:56.977653 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:56.977653 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:19:56.977653 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:19:56.977653 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:56.977653 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:19:56.977653 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:56.977653 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:19:56.978378 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:56.978378 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:19:56.978378 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:56.978378 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:19:56.978378 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:19:56.978378 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:19:56.978378 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 15 10:19:56.978378 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:19:56.978378 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:19:56.978378 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:56.978378 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:56.978759 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:56.978759 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:19:56.978759 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:56.978759 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:56.978759 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:56.978759 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:19:56.978759 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:56.978759 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:19:56.978759 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:19:56.978759 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:56.978759 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:19:56.978759 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:56.978759 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:19:56.978759 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:19:56.979126 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:56.979126 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:19:56.979126 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:56.979126 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:19:56.979691 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:56.979691 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:19:56.979691 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:19:56.979691 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:19:56.979691 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:19:56.979691 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:19:56.979691 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:56.979691 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:19:56.979691 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:19:56.979691 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:56.979691 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:19:56.979691 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:19:56.979691 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:56.979691 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:56.979691 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:19:56.979691 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:56.979691 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:19:56.980190 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 15 10:19:56.980190 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:19:56.980190 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:19:56.980190 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:56.980190 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:56.980190 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:56.980190 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:19:56.980190 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:56.980190 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:56.980190 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:56.980190 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:19:56.980190 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:56.980190 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:19:56.980190 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:19:56.980190 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:56.980190 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:19:56.980190 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:56.980190 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:19:56.980190 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:19:56.980722 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:56.980722 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:19:56.980722 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:56.980722 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:19:56.985005 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:19:56.985297 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:19:56.985436 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:19:56.985659 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:19:56.986066 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:19:56.986324 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:19:56.986443 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:56.986987 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:19:56.986987 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:56.986987 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:19:56.986987 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:19:56.986987 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:56.986987 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:19:56.986987 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:56.986987 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:19:56.988054 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:56.988054 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:19:56.988054 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:56.988054 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:19:56.988054 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:19:56.988054 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:19:56.988054 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:19:56.988054 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:56.988054 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:56.988054 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:56.988054 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:19:56.988054 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:56.988430 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:19:56.988430 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:56.988430 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:19:56.988430 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:56.988430 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:19:56.988430 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:19:56.988430 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:56.988430 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:19:56.988430 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:56.988430 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:19:56.988430 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:19:56.988430 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:56.988430 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:19:56.988802 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:56.988802 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:19:56.989268 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:56.989268 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:19:56.989268 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:19:56.989268 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:19:56.989268 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:19:56.989268 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:19:56.989268 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:56.989268 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:19:56.989268 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:19:56.989268 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:56.989268 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:19:56.989268 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:19:56.989268 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:19:56.989268 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:19:56.989268 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:19:56.989268 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:19:56.989756 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:19:56.989756 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:19:56.989756 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:56.989756 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:56.989756 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:19:56.989756 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:19:56.989756 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:19:56.989756 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:19:56.989756 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:19:56.989756 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:19:56.989756 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:19:56.989756 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:19:56.989756 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:19:56.989756 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:19:56.989756 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:19:56.989756 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:19:56.989756 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:19:56.989756 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:19:56.989756 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:19:56.990275 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:19:56.990275 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:19:56.990275 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:20:01.406587 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:20:01.407214 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:20:01.407441 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:20:01.410122 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:20:01.449615 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:20:01.450103 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:01.451062 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:20:01.451062 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:01.451062 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:20:01.451062 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:20:01.451062 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:01.451062 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:20:01.451062 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:01.451062 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:20:01.452180 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_fdb_aging_settings failed after 0.044 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:01.452180 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:20:01.452180 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:01.452180 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:20:01.452180 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:20:01.452180 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:20:01.452180 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:20:01.452180 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:01.452180 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:01.452180 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:01.452180 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:20:01.452180 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:01.453010 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:01.453010 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:01.453010 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:20:01.453010 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:01.453010 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:20:01.453010 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:20:01.453010 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:01.453010 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:20:01.453010 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:01.453010 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:20:01.453010 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:20:01.453010 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:01.453010 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:20:01.453513 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:01.453513 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:20:01.453513 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:01.453513 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:20:01.453513 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:20:01.453513 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:20:01.453513 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:20:01.453513 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:20:01.453513 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:01.453513 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:20:01.453513 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:20:01.453513 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:01.453513 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:20:01.453513 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:20:01.453513 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:01.454115 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:01.454115 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:20:01.454115 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:01.454115 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:20:01.454115 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:20:01.454115 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:01.454115 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:01.454115 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:01.454115 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:20:01.454115 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:01.454115 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:01.454115 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:01.454115 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:20:01.454115 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:01.454115 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:20:01.454115 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:20:01.454115 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:01.454115 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:20:01.454115 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:01.454871 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:20:01.454871 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:20:01.454871 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:01.454871 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:20:01.454871 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:01.454871 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:20:01.951310 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:20:01.952260 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:20:01.952260 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:20:01.953268 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b027d63e-13a5-445b-84a0-8ae6223e05d5, routes=[], if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:20:01.953699 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:01.954799 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:20:01.954799 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:01.954799 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:20:01.954799 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:20:01.954799 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:01.954799 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:20:01.954799 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:01.954799 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:20:01.955927 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:01.955927 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:20:01.955927 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:01.955927 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:20:01.955927 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:20:01.955927 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:20:01.955927 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:20:01.955927 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:01.955927 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:01.955927 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:01.955927 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:20:01.955927 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:01.956879 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:01.956879 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:01.956879 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:20:01.956879 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:01.956879 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:20:01.956879 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:20:01.956879 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:01.956879 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:20:01.956879 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:01.956879 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:20:01.956879 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:20:01.956879 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:01.956879 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:20:01.957667 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:01.957667 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:20:01.957667 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:01.957667 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:20:01.957667 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:20:01.957667 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:20:01.957667 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:20:01.957667 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:20:01.957667 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:01.957667 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:20:01.957667 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:20:01.957667 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:01.957667 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:20:01.957667 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:20:01.957667 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:01.958517 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:01.958517 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:20:01.958517 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:01.958517 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:20:01.958517 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:20:01.958517 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:01.958517 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:01.958517 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:01.958517 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:20:01.958517 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:01.958517 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:01.958517 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:01.958517 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:20:01.958517 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:01.958517 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:20:01.958517 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:20:01.958517 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:01.958517 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:20:01.958517 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:01.959993 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:20:01.959993 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:20:01.959993 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:01.959993 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:20:01.959993 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:01.959993 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:20:01.961573 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:20:01.961957 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:20:01.962088 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:20:01.962235 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105458) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 15 10:20:01.962829 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:20:01.963151 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:01.963941 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:20:01.963941 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:01.963941 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:20:01.963941 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:20:01.963941 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:01.963941 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:20:01.963941 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:01.963941 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:20:01.964872 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:01.964872 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:20:01.964872 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:20:01.964872 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:20:01.964872 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:01.964872 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:20:01.964872 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:01.964872 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:20:01.964872 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:01.964872 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:20:01.964872 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:01.964872 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:20:01.964872 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:01.964872 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:20:01.964872 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:20:01.965507 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:01.965507 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:20:01.965507 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:01.965507 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:20:01.965507 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:20:01.965507 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:01.965507 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:20:01.965507 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:01.965507 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:20:01.965856 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:01.965856 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:20:01.965856 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:01.965856 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:20:01.965856 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:20:01.965856 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:20:01.965856 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:20:01.965856 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 15 10:20:01.965856 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:20:01.965856 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:20:01.965856 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:01.965856 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:01.966578 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:01.966578 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:20:01.966578 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:01.966578 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:01.966578 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:01.966578 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:20:01.966578 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:01.966578 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:20:01.966578 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:20:01.966578 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:01.966578 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:20:01.966578 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:01.966578 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:20:01.966578 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:20:01.967149 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:01.967149 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:20:01.967149 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:01.967149 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:20:01.967149 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:01.967149 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:20:01.967149 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:20:01.967149 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:20:01.967149 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:20:01.967149 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:20:01.967149 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:01.967149 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:20:01.967149 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:20:01.967149 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:01.967895 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:20:01.967895 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:20:01.967895 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:01.967895 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:01.967895 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:20:01.967895 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:01.967895 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:20:01.967895 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:20:01.967895 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^ May 15 10:20:01.967895 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:20:01.967895 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:20:01.967895 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:01.967895 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:01.967895 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:01.967895 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:20:01.967895 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:01.967895 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:01.967895 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:01.967895 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:20:01.969348 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:01.969348 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:20:01.969348 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:20:01.969348 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:01.969348 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:20:01.969348 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:01.969348 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:20:01.969348 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:20:01.969348 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:01.969348 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:20:01.969348 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:01.969348 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:20:01.974902 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:20:01.975328 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_ha_failover {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:20:01.975452 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:20:01.975594 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:20:01.976082 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:20:01.976499 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:01.977269 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:20:01.977269 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:01.977269 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:20:01.977269 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:20:01.977269 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:01.977269 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:20:01.977269 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:01.977269 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:20:01.978225 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:01.978225 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:20:01.978225 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:20:01.978225 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:20:01.978225 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:01.978225 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:20:01.978225 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:01.978225 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:20:01.978225 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:01.978225 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:20:01.978225 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:01.978225 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:20:01.978225 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:01.978225 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:20:01.978225 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:20:01.980752 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:01.980752 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:20:01.980752 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:01.980752 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:20:01.980752 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:20:01.980752 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:01.980752 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:20:01.980752 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:01.980752 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:20:01.980752 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:01.980752 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:20:01.980752 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:01.980752 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:20:01.980752 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:20:01.981183 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:20:01.981183 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 15 10:20:01.981183 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:20:01.981183 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:20:01.981183 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:01.981183 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:01.981183 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:01.981183 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:20:01.981183 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:01.981183 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:01.981183 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:01.981183 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:20:01.981183 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:01.981542 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:20:01.981542 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:20:01.981542 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:01.981542 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:20:01.981542 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:01.981542 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:20:01.981542 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:20:01.981542 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:01.981542 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:20:01.981542 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:01.981542 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:20:01.981542 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:01.982180 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:20:01.982180 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:20:01.982180 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:20:01.982180 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:20:01.982180 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:20:01.982180 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:01.982180 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:20:01.982180 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:20:01.982180 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:01.982180 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:20:01.982180 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:20:01.982180 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:01.982180 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:01.982180 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:20:01.982180 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:01.982180 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:20:01.982180 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 15 10:20:01.982180 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:20:01.982180 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:20:01.982759 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:01.982759 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:01.982759 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:01.982759 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:20:01.982759 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:01.982759 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:01.982759 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:01.982759 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:20:01.982759 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:01.982759 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:20:01.982759 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:20:01.982759 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:01.982759 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:20:01.982759 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:01.982759 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:20:01.982759 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:20:01.982759 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:01.982759 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:20:01.982759 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:01.982759 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:20:01.985495 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:20:01.985848 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:20:01.985975 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:20:01.986152 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:20:01.986461 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:20:01.986781 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:20:01.986918 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:01.987418 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:20:01.987418 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:01.987418 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:20:01.987418 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:20:01.987418 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:01.987418 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:20:01.987418 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:01.987418 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:20:01.988307 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:01.988307 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:20:01.988307 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:01.988307 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:20:01.988307 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:20:01.988307 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:20:01.988307 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:20:01.988307 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:01.988307 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:01.988307 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:01.988307 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:20:01.988307 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:01.988728 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:01.988728 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:01.988728 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:20:01.988728 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:01.988728 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:20:01.988728 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:20:01.988728 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:01.988728 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:20:01.988728 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:01.988728 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:20:01.988728 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:20:01.988728 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:01.988728 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:20:01.989517 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:01.989517 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:20:01.989517 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:01.989517 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:20:01.989517 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:20:01.989517 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:20:01.989517 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:20:01.989517 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:20:01.989517 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:01.989517 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:20:01.989517 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:20:01.989517 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:01.989517 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:20:01.989517 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:20:01.989517 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:01.989985 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:01.989985 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:20:01.989985 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:01.989985 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:20:01.989985 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:20:01.989985 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:01.989985 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:01.989985 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:01.989985 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:20:01.989985 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:01.989985 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:01.989985 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:01.989985 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:20:01.989985 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:01.989985 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:20:01.989985 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:20:01.989985 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:01.989985 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:20:01.989985 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:01.990745 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:20:01.990745 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:20:01.990745 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:01.990745 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:20:01.990745 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:01.990745 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:20:06.407605 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:20:06.408563 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:20:06.408563 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:20:06.409437 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:20:06.442135 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:20:06.442547 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:06.443647 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:20:06.443647 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:06.443647 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:20:06.443647 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:20:06.443647 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:06.443647 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:20:06.443647 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:06.443647 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:20:06.444539 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_fdb_aging_settings failed after 0.036 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:06.444539 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:20:06.444539 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:06.444539 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:20:06.444539 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:20:06.444539 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:20:06.444539 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:20:06.444539 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:06.444539 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:06.444539 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:06.444539 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:20:06.444539 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:06.445100 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:06.445100 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:06.445100 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:20:06.445100 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:06.445100 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:20:06.445100 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:20:06.445100 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:06.445100 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:20:06.445100 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:06.445100 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:20:06.445100 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:20:06.445100 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:06.445100 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:20:06.445383 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:06.445383 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:20:06.445634 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:06.445634 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:20:06.445634 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:20:06.445634 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:20:06.445634 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:20:06.445634 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:20:06.445634 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:06.445634 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:20:06.445634 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:20:06.445634 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:06.445634 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:20:06.445634 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:20:06.445634 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:06.445634 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:06.445634 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:20:06.445634 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:06.445971 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:20:06.445971 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:20:06.445971 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:06.445971 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:06.445971 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:06.445971 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:20:06.445971 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:06.445971 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:06.445971 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:06.445971 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:20:06.445971 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:06.445971 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:20:06.445971 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:20:06.445971 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:06.445971 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:20:06.445971 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:06.445971 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:20:06.445971 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:20:06.445971 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:06.446359 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:20:06.446359 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:06.446359 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:20:06.952118 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:20:06.953174 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:20:06.953377 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:20:06.955115 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b027d63e-13a5-445b-84a0-8ae6223e05d5, routes=[], if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:20:06.955494 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:06.956208 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:20:06.956208 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:06.956208 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:20:06.956208 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:20:06.956208 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:06.956208 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:20:06.956208 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:06.956208 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:20:06.957543 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:06.957543 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:20:06.957543 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:06.957543 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:20:06.957543 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:20:06.957543 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:20:06.957543 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:20:06.957543 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:06.957543 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:06.957543 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:06.957543 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:20:06.957543 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:06.958279 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:06.958279 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:06.958279 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:20:06.958279 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:06.958279 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:20:06.958279 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:20:06.958279 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:06.958279 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:20:06.958279 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:06.958279 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:20:06.958279 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:20:06.958279 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:06.958279 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:20:06.959018 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:06.959018 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:20:06.959018 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:06.959018 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:20:06.959018 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:20:06.959018 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:20:06.959018 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:20:06.959018 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:20:06.959018 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:06.959018 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:20:06.959018 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:20:06.959018 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:06.959018 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:20:06.959018 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:20:06.959018 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:06.959525 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:06.959525 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:20:06.959525 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:06.959525 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:20:06.959525 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:20:06.959525 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:06.959525 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:06.959525 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:06.959525 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:20:06.959525 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:06.959525 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:06.959525 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:06.959525 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:20:06.959525 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:06.959525 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:20:06.959525 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:20:06.959525 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:06.959525 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:20:06.959525 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:06.960203 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:20:06.960203 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:20:06.960203 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:06.960203 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:20:06.960203 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:06.960203 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:20:06.962089 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:20:06.962420 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:20:06.962596 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:20:06.962765 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105458) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 15 10:20:06.963274 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:20:06.963532 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:06.964321 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:20:06.964321 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:06.964321 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:20:06.964321 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:20:06.964321 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:06.964321 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:20:06.964321 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:06.964321 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:20:06.965182 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:06.965182 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:20:06.965182 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:20:06.965182 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:20:06.965182 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:06.965182 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:20:06.965182 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:06.965182 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:20:06.965182 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:06.965182 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:20:06.965182 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:06.965182 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:20:06.965182 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:06.965182 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:20:06.965182 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:20:06.965759 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:06.965759 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:20:06.965759 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:06.965759 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:20:06.965759 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:20:06.965759 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:06.965759 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:20:06.965759 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:06.965759 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:20:06.966051 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:06.966051 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:20:06.966051 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:06.966051 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:20:06.966051 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:20:06.966051 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:20:06.966051 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:20:06.966051 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 15 10:20:06.966051 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:20:06.966051 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:20:06.966051 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:06.966051 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:06.966470 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:06.966470 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:20:06.966470 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:06.966470 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:06.966470 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:06.966470 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:20:06.966470 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:06.966470 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:20:06.966470 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:20:06.966470 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:06.966470 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:20:06.966470 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:06.966470 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:20:06.966470 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:20:06.966997 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:06.966997 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:20:06.966997 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:06.966997 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:20:06.967138 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:06.967138 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:20:06.967138 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:20:06.967138 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:20:06.967138 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:20:06.967138 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:20:06.967138 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:06.967138 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:20:06.967138 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:20:06.967138 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:06.967138 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:20:06.967138 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:20:06.967138 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:06.967138 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:06.967138 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:20:06.967138 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:06.967644 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:20:06.967644 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:20:06.967644 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^ May 15 10:20:06.967644 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:20:06.967644 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:20:06.967644 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:06.967644 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:06.967644 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:06.967644 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:20:06.967644 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:06.967644 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:06.967644 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:06.967644 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:20:06.967644 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:06.967644 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:20:06.967644 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:20:06.967644 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:06.967644 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:20:06.968213 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:06.968213 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:20:06.968213 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:20:06.968213 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:06.968213 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:20:06.968213 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:06.968213 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:20:06.975333 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:20:06.975811 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_ha_failover {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:20:06.976027 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:20:06.976266 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:20:06.977284 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:20:06.977767 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:06.978950 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:20:06.978950 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:06.978950 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:20:06.978950 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:20:06.978950 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:06.978950 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:20:06.978950 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:06.978950 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:20:06.980461 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:06.980461 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:20:06.980461 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:20:06.980461 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:20:06.980461 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:06.980461 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:20:06.980461 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:06.980461 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:20:06.980461 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:06.980461 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:20:06.980461 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:06.980461 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:20:06.980461 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:06.980461 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:20:06.980461 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:20:06.980986 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:06.980986 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:20:06.980986 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:06.980986 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:20:06.980986 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:20:06.980986 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:06.980986 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:20:06.980986 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:06.980986 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:20:06.982218 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_ha_failover failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:06.982218 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:20:06.982218 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:06.982218 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:20:06.982218 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:20:06.982218 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:20:06.982218 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 15 10:20:06.982218 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:20:06.982218 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:20:06.982218 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:06.982218 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:06.982703 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:06.982703 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:20:06.982703 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:06.982703 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:06.982703 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:06.982703 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:20:06.982703 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:06.982703 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:20:06.982703 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:20:06.982703 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:06.982703 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:20:06.982703 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:06.982703 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:20:06.982703 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:20:06.983256 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:06.983256 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:20:06.983256 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:06.983256 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:20:06.983736 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:06.983736 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:20:06.983736 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:20:06.983736 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:20:06.983736 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:20:06.983736 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:20:06.983736 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:06.983736 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:20:06.983736 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:20:06.983736 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:06.983736 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:20:06.983736 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:20:06.983736 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:06.983736 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:06.983736 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:20:06.983736 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:06.983736 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:20:06.984332 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 15 10:20:06.984332 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:20:06.984332 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:20:06.984332 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:06.984332 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:06.984332 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:06.984332 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:20:06.984332 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:06.984332 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:06.984332 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:06.984332 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:20:06.984332 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:06.984332 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:20:06.984332 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:20:06.984332 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:06.984332 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:20:06.984332 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:06.984332 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:20:06.984332 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:20:06.985015 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:06.985015 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:20:06.985015 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:06.985015 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:20:06.986018 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:20:06.986489 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:20:06.987122 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:20:06.987463 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:20:06.988271 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:20:06.988801 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:20:06.989074 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:06.990117 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:20:06.990117 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:06.990117 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:20:06.990117 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:20:06.990117 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:06.990117 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:20:06.990117 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:06.990117 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:20:06.991555 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:06.991555 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:20:06.991555 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:06.991555 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:20:06.991555 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:20:06.991555 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:20:06.991555 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:20:06.991555 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:06.991555 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:06.991555 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:06.991555 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:20:06.991555 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:06.991996 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:06.991996 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:06.991996 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:20:06.991996 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:06.991996 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:20:06.991996 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:20:06.991996 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:06.991996 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:20:06.991996 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:06.991996 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:20:06.991996 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:20:06.991996 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:06.991996 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:20:06.992367 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:06.992367 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:20:06.992847 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:06.992847 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:20:06.992847 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:20:06.992847 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:20:06.992847 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:20:06.992847 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:20:06.992847 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:06.992847 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:20:06.992847 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:20:06.992847 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:06.992847 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:20:06.992847 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:20:06.992847 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:06.992847 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:06.992847 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:20:06.992847 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:06.993457 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:20:06.993457 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:20:06.993457 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:06.993457 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:06.993457 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:06.993457 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:20:06.993457 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:06.993457 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:06.993457 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:06.993457 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:20:06.993457 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:06.993457 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:20:06.993457 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:20:06.993457 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:06.993457 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:20:06.993457 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:06.993457 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:20:06.993457 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:20:06.993457 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:06.994219 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:20:06.994219 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:06.994219 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:20:11.408228 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:20:11.409167 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:20:11.409167 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:20:11.410110 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:20:11.444048 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:20:11.444557 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:11.445693 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:20:11.445693 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:11.445693 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:20:11.445693 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:20:11.445693 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:11.445693 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:20:11.445693 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:11.445693 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:20:11.447620 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_fdb_aging_settings failed after 0.037 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:11.447620 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:20:11.447620 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:11.447620 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:20:11.447620 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:20:11.447620 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:20:11.447620 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:20:11.447620 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:11.447620 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:11.447620 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:11.447620 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:20:11.447620 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:11.448192 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:11.448192 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:11.448192 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:20:11.448192 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:11.448192 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:20:11.448192 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:20:11.448192 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:11.448192 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:20:11.448192 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:11.448192 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:20:11.448192 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:20:11.448192 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:11.448192 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:20:11.448599 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:11.448599 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:20:11.449522 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:11.449522 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:20:11.449522 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:20:11.449522 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:20:11.449522 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:20:11.449522 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:20:11.449522 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:11.449522 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:20:11.449522 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:20:11.449522 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:11.449522 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:20:11.449522 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:20:11.449522 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:11.449522 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:11.449522 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:20:11.449522 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:11.450058 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:20:11.450058 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:20:11.450058 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:11.450058 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:11.450058 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:11.450058 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:20:11.450058 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:11.450058 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:11.450058 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:11.450058 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:20:11.450058 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:11.450058 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:20:11.450058 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:20:11.450058 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:11.450058 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:20:11.450058 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:11.450058 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:20:11.450058 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:20:11.450058 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:11.450618 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:20:11.450618 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:11.450618 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:20:11.953026 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:20:11.953568 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:20:11.953690 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:20:11.954326 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b027d63e-13a5-445b-84a0-8ae6223e05d5, routes=[], if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:20:11.954501 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:11.955081 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:20:11.955081 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:11.955081 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:20:11.955081 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:20:11.955081 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:11.955081 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:20:11.955081 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:11.955081 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:20:11.956018 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:11.956018 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:20:11.956018 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:11.956018 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:20:11.956018 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:20:11.956018 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:20:11.956018 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:20:11.956018 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:11.956018 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:11.956018 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:11.956018 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:20:11.956018 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:11.956480 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:11.956480 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:11.956480 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:20:11.956480 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:11.956480 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:20:11.956480 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:20:11.956480 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:11.956480 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:20:11.956480 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:11.956480 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:20:11.956480 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:20:11.956480 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:11.956480 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:20:11.956971 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:11.956971 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:20:11.957085 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:11.957085 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:20:11.957085 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:20:11.957085 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:20:11.957085 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:20:11.957085 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:20:11.957085 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:11.957085 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:20:11.957085 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:20:11.957085 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:11.957085 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:20:11.957085 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:20:11.957085 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:11.957085 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:11.957085 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:20:11.957085 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:11.957611 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:20:11.957611 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:20:11.957611 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:11.957611 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:11.957611 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:11.957611 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:20:11.957611 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:11.957611 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:11.957611 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:11.957611 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:20:11.957611 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:11.957611 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:20:11.957611 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:20:11.957611 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:11.957611 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:20:11.957611 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:11.957611 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:20:11.957611 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:20:11.957611 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:11.958151 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:20:11.958151 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:11.958151 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:20:11.962689 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:20:11.963014 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:20:11.963126 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:20:11.963239 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105458) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 15 10:20:11.963499 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:20:11.963667 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:11.964126 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:20:11.964126 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:11.964126 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:20:11.964126 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:20:11.964126 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:11.964126 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:20:11.964126 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:11.964126 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:20:11.964875 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:11.964875 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:20:11.964875 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:20:11.964875 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:20:11.964875 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:11.964875 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:20:11.964875 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:11.964875 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:20:11.964875 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:11.964875 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:20:11.964875 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:11.964875 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:20:11.964875 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:11.964875 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:20:11.964875 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:20:11.965403 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:11.965403 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:20:11.965403 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:11.965403 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:20:11.965403 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:20:11.965403 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:11.965403 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:20:11.965403 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:11.965403 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:20:11.965773 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:11.965773 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:20:11.965773 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:11.965773 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:20:11.965773 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:20:11.965773 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:20:11.965773 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:20:11.965773 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 15 10:20:11.965773 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:20:11.965773 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:20:11.965773 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:11.965773 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:11.966157 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:11.966157 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:20:11.966157 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:11.966157 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:11.966157 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:11.966157 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:20:11.966157 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:11.966157 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:20:11.966157 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:20:11.966157 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:11.966157 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:20:11.966157 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:11.966157 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:20:11.966157 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:20:11.966606 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:11.966606 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:20:11.966606 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:11.966606 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:20:11.966848 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:11.966848 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:20:11.966848 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:20:11.966848 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:20:11.966848 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:20:11.966848 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:20:11.966848 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:11.966848 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:20:11.966848 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:20:11.966848 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:11.966848 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:20:11.966848 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:20:11.966848 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:11.966848 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:11.966848 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:20:11.966848 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:11.967307 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:20:11.967307 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:20:11.967307 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^ May 15 10:20:11.967307 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:20:11.967307 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:20:11.967307 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:11.967307 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:11.967307 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:11.967307 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:20:11.967307 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:11.967307 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:11.967307 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:11.967307 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:20:11.967307 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:11.967307 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:20:11.967307 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:20:11.967307 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:11.967307 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:20:11.967888 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:11.967888 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:20:11.967888 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:20:11.967888 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:11.967888 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:20:11.967888 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:11.967888 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:20:11.975917 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:20:11.976231 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_ha_failover {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:20:11.976332 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:20:11.976465 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:20:11.976756 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:20:11.976979 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:11.977487 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:20:11.977487 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:11.977487 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:20:11.977487 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:20:11.977487 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:11.977487 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:20:11.977487 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:11.977487 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:20:11.978189 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:11.978189 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:20:11.978189 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:20:11.978189 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:20:11.978189 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:11.978189 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:20:11.978189 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:11.978189 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:20:11.978189 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:11.978189 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:20:11.978189 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:11.978189 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:20:11.978189 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:11.978189 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:20:11.978189 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:20:11.979076 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:11.979076 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:20:11.979076 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:11.979076 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:20:11.979076 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:20:11.979076 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:11.979076 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:20:11.979076 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:11.979076 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:20:11.979076 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:11.979076 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:20:11.979076 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:11.979076 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:20:11.979076 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:20:11.979918 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:20:11.979918 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 15 10:20:11.979918 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:20:11.979918 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:20:11.979918 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:11.979918 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:11.979918 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:11.979918 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:20:11.979918 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:11.979918 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:11.979918 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:11.979918 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:20:11.979918 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:11.980563 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:20:11.980563 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:20:11.980563 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:11.980563 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:20:11.980563 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:11.980563 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:20:11.980563 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:20:11.980563 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:11.980563 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:20:11.980563 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:11.980563 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:20:11.980563 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:11.981297 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:20:11.981297 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:20:11.981297 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:20:11.981297 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:20:11.981297 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:20:11.981297 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:11.981297 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:20:11.981297 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:20:11.981297 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:11.981297 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:20:11.981297 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:20:11.981297 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:11.981297 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:11.981297 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:20:11.981297 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:11.981297 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:20:11.981297 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 15 10:20:11.981297 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:20:11.981297 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:20:11.982410 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:11.982410 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:11.982410 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:11.982410 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:20:11.982410 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:11.982410 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:11.982410 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:11.982410 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:20:11.982410 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:11.982410 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:20:11.982410 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:20:11.982410 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:11.982410 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:20:11.982410 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:11.982410 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:20:11.982410 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:20:11.982410 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:11.982410 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:20:11.982410 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:11.982410 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:20:11.986411 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:20:11.986658 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:20:11.986817 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:20:11.986988 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:20:11.987213 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:20:11.987430 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:20:11.987524 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:11.988082 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:20:11.988082 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:11.988082 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:20:11.988082 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:20:11.988082 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:11.988082 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:20:11.988082 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:11.988082 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:20:11.988966 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:11.988966 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:20:11.988966 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:11.988966 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:20:11.988966 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:20:11.988966 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:20:11.988966 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:20:11.988966 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:11.988966 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:11.988966 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:11.988966 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:20:11.988966 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:11.989340 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:11.989340 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:11.989340 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:20:11.989340 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:11.989340 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:20:11.989340 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:20:11.989340 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:11.989340 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:20:11.989340 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:11.989340 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:20:11.989340 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:20:11.989340 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:11.989340 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:20:11.989749 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:11.989749 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:20:11.990071 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:11.990071 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:20:11.990071 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:20:11.990071 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:20:11.990071 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:20:11.990071 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:20:11.990071 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:11.990071 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:20:11.990071 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:20:11.990071 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:11.990071 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:20:11.990071 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:20:11.990071 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:11.990071 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:11.990071 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:20:11.990071 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:11.990517 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:20:11.990517 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:20:11.990517 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:11.990517 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:11.990517 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:11.990517 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:20:11.990517 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:11.990517 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:11.990517 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:11.990517 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:20:11.990517 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:11.990517 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:20:11.990517 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:20:11.990517 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:11.990517 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:20:11.990517 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:11.990517 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:20:11.990517 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:20:11.990517 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:11.991096 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:20:11.991096 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:11.991096 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:20:16.409229 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:20:16.410141 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:20:16.410211 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:20:16.411577 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:20:16.453041 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:20:16.453390 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:16.454447 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:20:16.454447 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:16.454447 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:20:16.454447 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:20:16.454447 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:16.454447 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:20:16.454447 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:16.454447 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:20:16.456139 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_fdb_aging_settings failed after 0.044 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:16.456139 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:20:16.456139 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:16.456139 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:20:16.456139 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:20:16.456139 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:20:16.456139 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:20:16.456139 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:16.456139 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:16.456139 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:16.456139 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:20:16.456139 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:16.456581 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:16.456581 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:16.456581 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:20:16.456581 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:16.456581 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:20:16.456581 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:20:16.456581 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:16.456581 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:20:16.456581 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:16.456581 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:20:16.456581 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:20:16.456581 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:16.456581 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:20:16.456973 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:16.456973 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:20:16.458003 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:16.458003 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:20:16.458003 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:20:16.458003 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:20:16.458003 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:20:16.458003 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:20:16.458003 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:16.458003 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:20:16.458003 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:20:16.458003 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:16.458003 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:20:16.458003 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:20:16.458003 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:16.458003 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:16.458003 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:20:16.458003 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:16.458621 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:20:16.458621 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:20:16.458621 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:16.458621 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:16.458621 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:16.458621 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:20:16.458621 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:16.458621 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:16.458621 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:16.458621 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:20:16.458621 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:16.458621 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:20:16.458621 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:20:16.458621 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:16.458621 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:20:16.458621 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:16.458621 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:20:16.458621 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:20:16.458621 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:16.459237 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:20:16.459237 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:16.459237 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:20:16.953944 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:20:16.954718 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:20:16.955038 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:20:16.956136 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b027d63e-13a5-445b-84a0-8ae6223e05d5, routes=[], if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:20:16.956452 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:16.957350 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:20:16.957350 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:16.957350 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:20:16.957350 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:20:16.957350 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:16.957350 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:20:16.957350 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:16.957350 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:20:16.959106 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:16.959106 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:20:16.959106 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:16.959106 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:20:16.959106 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:20:16.959106 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:20:16.959106 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:20:16.959106 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:16.959106 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:16.959106 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:16.959106 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:20:16.959106 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:16.959945 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:16.959945 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:16.959945 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:20:16.959945 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:16.959945 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:20:16.959945 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:20:16.959945 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:16.959945 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:20:16.959945 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:16.959945 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:20:16.959945 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:20:16.959945 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:16.959945 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:20:16.961454 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:16.961454 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:20:16.961454 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:16.961454 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:20:16.961454 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:20:16.961454 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:20:16.961454 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:20:16.961454 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:20:16.961454 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:16.961454 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:20:16.961454 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:20:16.961454 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:16.961454 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:20:16.961454 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:20:16.961454 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:16.963106 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:16.963106 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:20:16.963106 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:16.963106 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:20:16.963106 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:20:16.963106 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:16.963106 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:16.963106 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:16.963106 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:20:16.963106 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:16.963106 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:16.963106 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:16.963106 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:20:16.963106 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:16.963106 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:20:16.963106 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:20:16.963106 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:16.963106 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:20:16.963106 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:16.963938 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:20:16.963938 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:20:16.963938 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:16.963938 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:20:16.963938 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:16.963938 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:20:16.963938 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:20:16.963938 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:20:16.963938 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:20:16.963938 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105458) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 15 10:20:16.964433 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:20:16.964433 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:16.965157 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:20:16.965157 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:16.965157 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:20:16.965157 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:20:16.965157 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:16.965157 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:20:16.965157 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:16.965157 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:20:16.965750 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:16.965750 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:20:16.965750 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:20:16.965750 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:20:16.965750 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:16.965750 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:20:16.965750 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:16.965750 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:20:16.965750 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:16.965750 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:20:16.965750 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:16.965750 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:20:16.965750 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:16.965750 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:20:16.965750 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:20:16.966436 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:16.966436 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:20:16.966436 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:16.966436 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:20:16.966436 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:20:16.966436 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:16.966436 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:20:16.966436 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:16.966436 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:20:16.966436 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:16.966436 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:20:16.966436 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:16.966436 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:20:16.966436 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:20:16.966988 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:20:16.966988 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:20:16.966988 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 15 10:20:16.966988 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:20:16.966988 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:20:16.966988 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:16.966988 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:16.966988 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:16.966988 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:20:16.966988 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:16.966988 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:16.966988 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:16.966988 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:20:16.967456 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:16.967456 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:20:16.967456 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:20:16.967456 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:16.967456 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:20:16.967456 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:16.967456 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:20:16.967456 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:20:16.967456 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:16.967456 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:20:16.967456 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:16.967456 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:20:16.967896 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:16.967896 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:20:16.967896 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:20:16.967896 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:20:16.967896 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:20:16.967896 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:20:16.967896 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:16.967896 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:20:16.967896 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:20:16.967896 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:16.967896 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:20:16.967896 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:20:16.967896 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:16.967896 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:16.967896 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:20:16.967896 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:16.968337 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:20:16.968337 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:20:16.968337 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^ May 15 10:20:16.968337 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:20:16.968337 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:20:16.968337 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:16.968337 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:16.968337 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:16.968337 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:20:16.968337 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:16.968337 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:16.968337 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:16.968337 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:20:16.968337 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:16.968337 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:20:16.968337 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:20:16.968337 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:16.968337 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:20:16.968849 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:16.968849 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:20:16.968849 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:20:16.968849 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:16.968849 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:20:16.968849 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:16.968849 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:20:16.976325 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:20:16.976537 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_ha_failover {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:20:16.976623 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:20:16.976764 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:20:16.977496 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:20:16.977958 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:16.978747 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:20:16.978747 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:16.978747 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:20:16.978747 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:20:16.978747 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:16.978747 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:20:16.978747 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:16.978747 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:20:16.979401 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:16.979401 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:20:16.979401 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:20:16.979401 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:20:16.979401 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:16.979401 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:20:16.979401 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:16.979401 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:20:16.979401 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:16.979401 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:20:16.979401 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:16.979401 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:20:16.979401 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:16.979401 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:20:16.979401 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:20:16.980424 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:16.980424 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:20:16.980424 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:16.980424 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:20:16.980424 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:20:16.980424 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:16.980424 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:20:16.980424 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:16.980424 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:20:16.980424 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:16.980424 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:20:16.980424 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:16.980424 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:20:16.980424 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:20:16.981363 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:20:16.981363 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 15 10:20:16.981363 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:20:16.981363 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:20:16.981363 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:16.981363 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:16.981363 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:16.981363 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:20:16.981363 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:16.981363 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:16.981363 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:16.981363 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:20:16.981363 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:16.982176 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:20:16.982176 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:20:16.982176 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:16.982176 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:20:16.982176 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:16.982176 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:20:16.982176 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:20:16.982176 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:16.982176 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:20:16.982176 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:16.982176 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:20:16.982176 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:16.982983 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:20:16.982983 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:20:16.982983 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:20:16.982983 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:20:16.982983 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:20:16.982983 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:16.982983 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:20:16.982983 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:20:16.982983 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:16.982983 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:20:16.982983 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:20:16.982983 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:16.982983 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:16.982983 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:20:16.982983 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:16.982983 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:20:16.982983 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 15 10:20:16.982983 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:20:16.982983 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:20:16.984502 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:16.984502 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:16.984502 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:16.984502 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:20:16.984502 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:16.984502 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:16.984502 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:16.984502 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:20:16.984502 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:16.984502 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:20:16.984502 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:20:16.984502 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:16.984502 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:20:16.984502 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:16.984502 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:20:16.984502 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:20:16.984502 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:16.984502 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:20:16.984502 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:16.984502 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:20:16.986987 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:20:16.987548 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:20:16.987726 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:20:16.987925 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:20:16.988492 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:20:16.988955 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:20:16.988955 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:16.989524 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:20:16.989524 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:16.989524 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:20:16.989524 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:20:16.989524 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:16.989524 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:20:16.989524 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:16.989524 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:20:16.990560 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:16.990560 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:20:16.990560 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:16.990560 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:20:16.990560 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:20:16.990560 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:20:16.990560 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:20:16.990560 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:16.990560 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:16.990560 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:16.990560 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:20:16.990560 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:16.990914 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:16.990914 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:16.990914 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:20:16.990914 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:16.990914 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:20:16.990914 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:20:16.990914 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:16.990914 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:20:16.990914 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:16.990914 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:20:16.990914 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:20:16.990914 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:16.990914 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:20:16.991274 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:16.991274 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:20:16.991639 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:16.991639 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:20:16.991639 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:20:16.991639 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:20:16.991639 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:20:16.991639 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:20:16.991639 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:16.991639 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:20:16.991639 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:20:16.991639 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:16.991639 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:20:16.991639 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:20:16.991639 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:16.991639 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:16.991639 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:20:16.991639 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:16.992075 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:20:16.992075 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:20:16.992075 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:16.992075 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:16.992075 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:16.992075 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:20:16.992075 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:16.992075 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:16.992075 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:16.992075 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:20:16.992075 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:16.992075 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:20:16.992075 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:20:16.992075 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:16.992075 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:20:16.992075 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:16.992075 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:20:16.992075 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:20:16.992075 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:16.992611 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:20:16.992611 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:16.992611 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:20:21.409849 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:20:21.410784 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:20:21.410784 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:20:21.410784 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:20:21.444623 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:20:21.445096 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:21.446114 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:20:21.446114 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:21.446114 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:20:21.446114 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:20:21.446114 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:21.446114 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:20:21.446114 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:21.446114 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:20:21.447890 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_fdb_aging_settings failed after 0.036 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:21.447890 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:20:21.447890 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:21.447890 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:20:21.447890 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:20:21.447890 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:20:21.447890 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:20:21.447890 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:21.447890 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:21.447890 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:21.447890 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:20:21.447890 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:21.448511 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:21.448511 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:21.448511 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:20:21.448511 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:21.448511 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:20:21.448511 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:20:21.448511 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:21.448511 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:20:21.448511 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:21.448511 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:20:21.448511 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:20:21.448511 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:21.448511 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:20:21.449052 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:21.449052 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:20:21.449756 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:21.449756 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:20:21.449756 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:20:21.449756 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:20:21.449756 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:20:21.449756 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:20:21.449756 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:21.449756 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:20:21.449756 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:20:21.449756 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:21.449756 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:20:21.449756 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:20:21.449756 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:21.449756 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:21.449756 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:20:21.449756 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:21.450550 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:20:21.450550 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:20:21.450550 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:21.450550 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:21.450550 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:21.450550 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:20:21.450550 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:21.450550 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:21.450550 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:21.450550 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:20:21.450550 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:21.450550 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:20:21.450550 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:20:21.450550 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:21.450550 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:20:21.450550 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:21.450550 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:20:21.450550 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:20:21.450550 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:21.451575 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:20:21.451575 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:21.451575 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:20:21.954430 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:20:21.954992 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:20:21.955119 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:20:21.955910 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b027d63e-13a5-445b-84a0-8ae6223e05d5, routes=[], if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:20:21.956141 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:21.956791 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:20:21.956791 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:21.956791 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:20:21.956791 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:20:21.956791 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:21.956791 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:20:21.956791 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:21.956791 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:20:21.958235 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:21.958235 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:20:21.958235 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:21.958235 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:20:21.958235 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:20:21.958235 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:20:21.958235 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:20:21.958235 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:21.958235 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:21.958235 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:21.958235 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:20:21.958235 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:21.959087 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:21.959087 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:21.959087 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:20:21.959087 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:21.959087 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:20:21.959087 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:20:21.959087 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:21.959087 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:20:21.959087 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:21.959087 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:20:21.959087 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:20:21.959087 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:21.959087 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:20:21.959905 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:21.959905 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:20:21.959905 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:21.959905 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:20:21.959905 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:20:21.959905 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:20:21.959905 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:20:21.959905 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:20:21.959905 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:21.959905 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:20:21.959905 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:20:21.959905 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:21.959905 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:20:21.959905 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:20:21.959905 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:21.960876 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:21.960876 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:20:21.960876 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:21.960876 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:20:21.960876 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:20:21.960876 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:21.960876 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:21.960876 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:21.960876 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:20:21.960876 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:21.960876 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:21.960876 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:21.960876 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:20:21.960876 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:21.960876 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:20:21.960876 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:20:21.960876 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:21.960876 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:20:21.960876 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:21.961890 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:20:21.961890 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:20:21.961890 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:21.961890 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:20:21.961890 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:21.961890 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:20:21.963540 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:20:21.963951 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:20:21.964142 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:20:21.964343 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105458) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 15 10:20:21.964798 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:20:21.965060 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:21.965751 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:20:21.965751 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:21.965751 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:20:21.965751 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:20:21.965751 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:21.965751 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:20:21.965751 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:21.965751 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:20:21.966532 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:21.966532 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:20:21.966532 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:20:21.966532 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:20:21.966532 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:21.966532 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:20:21.966532 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:21.966532 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:20:21.966532 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:21.966532 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:20:21.966532 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:21.966532 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:20:21.966532 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:21.966532 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:20:21.966532 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:20:21.967154 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:21.967154 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:20:21.967154 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:21.967154 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:20:21.967154 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:20:21.967154 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:21.967154 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:20:21.967154 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:21.967154 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:20:21.967507 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:21.967507 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:20:21.967507 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:21.967507 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:20:21.967507 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:20:21.967507 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:20:21.967507 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:20:21.967507 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 15 10:20:21.967507 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:20:21.967507 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:20:21.967507 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:21.967507 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:21.968045 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:21.968045 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:20:21.968045 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:21.968045 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:21.968045 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:21.968045 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:20:21.968045 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:21.968045 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:20:21.968045 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:20:21.968045 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:21.968045 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:20:21.968045 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:21.968045 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:20:21.968045 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:20:21.968630 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:21.968630 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:20:21.968630 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:21.968630 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:20:21.968630 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:21.968630 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:20:21.968630 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:20:21.968630 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:20:21.968630 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:20:21.968630 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:20:21.968630 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:21.968630 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:20:21.968630 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:20:21.968630 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:21.969278 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:20:21.969278 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:20:21.969278 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:21.969278 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:21.969278 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:20:21.969278 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:21.969278 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:20:21.969278 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:20:21.969278 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^ May 15 10:20:21.969278 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:20:21.969278 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:20:21.969278 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:21.969278 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:21.969278 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:21.969278 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:20:21.969278 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:21.969278 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:21.969278 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:21.969278 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:20:21.970013 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:21.970013 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:20:21.970013 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:20:21.970013 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:21.970013 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:20:21.970013 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:21.970013 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:20:21.970013 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:20:21.970013 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:21.970013 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:20:21.970013 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:21.970013 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:20:21.977819 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:20:21.978295 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_ha_failover {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:20:21.978553 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:20:21.978919 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:20:21.979219 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:20:21.979486 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:21.980174 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:20:21.980174 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:21.980174 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:20:21.980174 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:20:21.980174 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:21.980174 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:20:21.980174 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:21.980174 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:20:21.981397 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:21.981397 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:20:21.981397 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:20:21.981397 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:20:21.981397 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:21.981397 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:20:21.981397 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:21.981397 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:20:21.981397 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:21.981397 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:20:21.981397 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:21.981397 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:20:21.981397 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:21.981397 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:20:21.981397 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:20:21.982013 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:21.982013 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:20:21.982013 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:21.982013 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:20:21.982013 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:20:21.982013 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:21.982013 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:20:21.982013 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:21.982013 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:20:21.982469 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:21.982469 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:20:21.982469 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:21.982469 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:20:21.982469 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:20:21.982469 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:20:21.982469 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 15 10:20:21.982469 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:20:21.982469 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:20:21.982469 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:21.982469 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:21.982939 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:21.982939 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:20:21.982939 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:21.982939 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:21.982939 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:21.982939 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:20:21.982939 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:21.982939 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:20:21.982939 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:20:21.982939 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:21.982939 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:20:21.982939 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:21.982939 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:20:21.982939 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:20:21.983443 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:21.983443 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:20:21.983443 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:21.983443 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:20:21.983762 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:21.983762 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:20:21.983762 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:20:21.983762 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:20:21.983762 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:20:21.983762 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:20:21.983762 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:21.983762 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:20:21.983762 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:20:21.983762 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:21.983762 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:20:21.983762 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:20:21.983762 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:21.983762 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:21.983762 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:20:21.983762 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:21.983762 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:20:21.984614 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 15 10:20:21.984614 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:20:21.984614 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:20:21.984614 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:21.984614 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:21.984614 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:21.984614 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:20:21.984614 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:21.984614 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:21.984614 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:21.984614 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:20:21.984614 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:21.984614 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:20:21.984614 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:20:21.984614 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:21.984614 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:20:21.984614 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:21.984614 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:20:21.984614 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:20:21.985511 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:21.985511 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:20:21.985511 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:21.985511 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:20:21.987653 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:20:21.988113 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:20:21.988240 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:20:21.988437 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:20:21.988848 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:20:21.989127 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:20:21.989301 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:21.989826 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:20:21.989826 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:21.989826 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:20:21.989826 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:20:21.989826 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:21.989826 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:20:21.989826 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:21.989826 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:20:21.990825 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:21.990825 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:20:21.990825 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:21.990825 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:20:21.990825 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:20:21.990825 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:20:21.990825 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:20:21.990825 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:21.990825 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:21.990825 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:21.990825 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:20:21.990825 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:21.991317 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:21.991317 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:21.991317 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:20:21.991317 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:21.991317 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:20:21.991317 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:20:21.991317 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:21.991317 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:20:21.991317 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:21.991317 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:20:21.991317 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:20:21.991317 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:21.991317 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:20:21.992101 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:21.992101 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:20:21.992101 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:21.992101 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:20:21.992101 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:20:21.992101 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:20:21.992101 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:20:21.992101 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:20:21.992101 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:21.992101 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:20:21.992101 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:20:21.992101 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:21.992101 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:20:21.992101 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:20:21.992101 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:21.992722 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:21.992722 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:20:21.992722 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:21.992722 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:20:21.992722 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:20:21.992722 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:21.992722 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:21.992722 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:21.992722 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:20:21.992722 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:21.992722 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:21.992722 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:21.992722 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:20:21.992722 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:21.992722 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:20:21.992722 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:20:21.992722 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:21.992722 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:20:21.992722 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:21.993476 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:20:21.993476 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:20:21.993476 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:21.993476 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:20:21.993476 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:21.993476 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:20:26.411265 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:20:26.412352 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:20:26.412539 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:20:26.413631 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:20:26.481902 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:20:26.482214 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:26.483212 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:20:26.483212 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:26.483212 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:20:26.483212 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:20:26.483212 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:26.483212 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:20:26.483212 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:26.483212 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:20:26.484596 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_fdb_aging_settings failed after 0.071 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:26.484596 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:20:26.484596 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:26.484596 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:20:26.484596 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:20:26.484596 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:20:26.484596 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:20:26.484596 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:26.484596 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:26.484596 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:26.484596 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:20:26.484596 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:26.485459 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:26.485459 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:26.485459 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:20:26.485459 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:26.485459 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:20:26.485459 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:20:26.485459 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:26.485459 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:20:26.485459 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:26.485459 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:20:26.485459 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:20:26.485459 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:26.485459 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:20:26.486024 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:26.486024 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:20:26.486024 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:26.486024 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:20:26.486024 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:20:26.486024 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:20:26.486024 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:20:26.486024 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:20:26.486024 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:26.486024 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:20:26.486024 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:20:26.486024 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:26.486024 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:20:26.486024 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:20:26.486024 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:26.486790 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:26.486790 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:20:26.486790 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:26.486790 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:20:26.486790 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:20:26.486790 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:26.486790 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:26.486790 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:26.486790 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:20:26.486790 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:26.486790 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:26.486790 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:26.486790 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:20:26.486790 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:26.486790 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:20:26.486790 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:20:26.486790 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:26.486790 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:20:26.486790 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:26.488451 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:20:26.488451 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:20:26.488451 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:26.488451 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:20:26.488451 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:26.488451 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:20:26.955121 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:20:26.955532 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:20:26.955868 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:20:26.957045 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b027d63e-13a5-445b-84a0-8ae6223e05d5, routes=[], if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:20:26.957380 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:26.958339 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:20:26.958339 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:26.958339 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:20:26.958339 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:20:26.958339 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:26.958339 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:20:26.958339 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:26.958339 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:20:26.960895 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:26.960895 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:20:26.960895 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:26.960895 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:20:26.960895 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:20:26.960895 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:20:26.960895 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:20:26.960895 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:26.960895 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:26.960895 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:26.960895 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:20:26.960895 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:26.961454 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:26.961454 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:26.961454 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:20:26.961454 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:26.961454 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:20:26.961454 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:20:26.961454 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:26.961454 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:20:26.961454 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:26.961454 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:20:26.961454 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:20:26.961454 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:26.961454 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:20:26.962014 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:26.962014 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:20:26.962795 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:26.962795 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:20:26.962795 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:20:26.962795 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:20:26.962795 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:20:26.962795 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:20:26.962795 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:26.962795 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:20:26.962795 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:20:26.962795 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:26.962795 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:20:26.962795 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:20:26.962795 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:26.962795 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:26.962795 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:20:26.962795 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:26.963778 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:20:26.963778 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:20:26.963778 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:26.963778 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:26.963778 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:26.963778 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:20:26.963778 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:26.963778 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:26.963778 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:26.963778 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:20:26.963778 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:26.963778 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:20:26.963778 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:20:26.963778 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:26.963778 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:20:26.963778 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:26.963778 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:20:26.963778 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:20:26.963778 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:26.964348 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:20:26.964348 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:26.964348 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:20:26.964348 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:20:26.964348 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:20:26.964348 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:20:26.964548 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105458) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 15 10:20:26.965039 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:20:26.965354 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:26.966482 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:20:26.966482 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:26.966482 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:20:26.966482 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:20:26.966482 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:26.966482 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:20:26.966482 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:26.966482 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:20:26.967200 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:26.967200 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:20:26.967200 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:20:26.967200 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:20:26.967200 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:26.967200 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:20:26.967200 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:26.967200 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:20:26.967200 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:26.967200 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:20:26.967200 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:26.967200 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:20:26.967200 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:26.967200 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:20:26.967200 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:20:26.968392 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:26.968392 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:20:26.968392 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:26.968392 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:20:26.968392 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:20:26.968392 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:26.968392 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:20:26.968392 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:26.968392 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:20:26.968392 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:26.968392 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:20:26.968392 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:26.968392 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:20:26.968392 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:20:26.969408 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:20:26.969408 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:20:26.969408 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 15 10:20:26.969408 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:20:26.969408 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:20:26.969408 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:26.969408 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:26.969408 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:26.969408 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:20:26.969408 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:26.969408 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:26.969408 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:26.969408 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:20:26.970312 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:26.970312 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:20:26.970312 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:20:26.970312 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:26.970312 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:20:26.970312 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:26.970312 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:20:26.970312 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:20:26.970312 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:26.970312 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:20:26.970312 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:26.970312 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:20:26.971210 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:26.971210 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:20:26.971210 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:20:26.971210 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:20:26.971210 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:20:26.971210 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:20:26.971210 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:26.971210 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:20:26.971210 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:20:26.971210 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:26.971210 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:20:26.971210 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:20:26.971210 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:26.971210 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:26.971210 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:20:26.971210 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:26.971875 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:20:26.971875 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:20:26.971875 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^ May 15 10:20:26.971875 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:20:26.971875 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:20:26.971875 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:26.971875 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:26.971875 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:26.971875 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:20:26.971875 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:26.971875 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:26.971875 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:26.971875 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:20:26.971875 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:26.971875 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:20:26.971875 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:20:26.971875 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:26.971875 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:20:26.972376 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:26.972376 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:20:26.972376 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:20:26.972376 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:26.972376 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:20:26.972376 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:26.972376 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:20:26.978225 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:20:26.978510 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_ha_failover {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:20:26.978655 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:20:26.978845 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:20:26.979555 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:20:26.979956 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:26.980650 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:20:26.980650 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:26.980650 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:20:26.980650 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:20:26.980650 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:26.980650 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:20:26.980650 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:26.980650 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:20:26.981755 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:26.981755 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:20:26.981755 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:20:26.981755 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:20:26.981755 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:26.981755 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:20:26.981755 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:26.981755 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:20:26.981755 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:26.981755 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:20:26.981755 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:26.981755 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:20:26.981755 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:26.981755 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:20:26.981755 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:20:26.982798 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:26.982798 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:20:26.982798 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:26.982798 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:20:26.982798 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:20:26.982798 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:26.982798 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:20:26.982798 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:26.982798 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:20:26.983385 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:26.983385 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:20:26.983385 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:26.983385 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:20:26.983385 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:20:26.983385 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:20:26.983385 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 15 10:20:26.983385 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:20:26.983385 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:20:26.983385 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:26.983385 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:26.984172 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:26.984172 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:20:26.984172 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:26.984172 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:26.984172 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:26.984172 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:20:26.984172 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:26.984172 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:20:26.984172 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:20:26.984172 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:26.984172 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:20:26.984172 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:26.984172 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:20:26.984172 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:20:26.985047 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:26.985047 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:20:26.985047 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:26.985047 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:20:26.985047 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:26.985047 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:20:26.985047 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:20:26.985047 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:20:26.985047 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:20:26.985047 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:20:26.985047 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:26.985047 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:20:26.985047 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:20:26.985047 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:26.985967 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:20:26.985967 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:20:26.985967 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:26.985967 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:26.985967 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:20:26.985967 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:26.985967 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:20:26.985967 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 15 10:20:26.985967 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:20:26.985967 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:20:26.985967 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:26.985967 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:26.985967 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:26.985967 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:20:26.985967 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:26.985967 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:26.985967 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:26.985967 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:20:26.985967 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:26.987138 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:20:26.987138 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:20:26.987138 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:26.987138 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:20:26.987138 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:26.987138 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:20:26.987138 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:20:26.987138 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:26.987138 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:20:26.987138 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:26.987138 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:20:26.988188 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:20:26.988421 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:20:26.988530 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:20:26.988616 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:20:26.988959 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:20:26.989177 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:20:26.989291 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:26.989883 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:20:26.989883 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:26.989883 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:20:26.989883 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:20:26.989883 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:26.989883 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:20:26.989883 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:26.989883 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:20:26.990956 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:26.990956 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:20:26.990956 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:26.990956 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:20:26.990956 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:20:26.990956 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:20:26.990956 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:20:26.990956 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:26.990956 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:26.990956 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:26.990956 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:20:26.990956 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:26.991433 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:26.991433 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:26.991433 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:20:26.991433 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:26.991433 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:20:26.991433 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:20:26.991433 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:26.991433 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:20:26.991433 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:26.991433 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:20:26.991433 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:20:26.991433 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:26.991433 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:20:26.991784 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:26.991784 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:20:26.992283 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:26.992283 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:20:26.992283 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:20:26.992283 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:20:26.992283 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:20:26.992283 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:20:26.992283 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:26.992283 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:20:26.992283 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:20:26.992283 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:26.992283 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:20:26.992283 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:20:26.992283 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:26.992283 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:26.992283 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:20:26.992283 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:26.992667 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:20:26.992667 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:20:26.992667 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:26.992667 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:26.992667 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:26.992667 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:20:26.992667 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:26.992667 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:26.992667 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:26.992667 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:20:26.992667 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:26.992667 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:20:26.992667 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:20:26.992667 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:26.992667 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:20:26.992667 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:26.992667 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:20:26.992667 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:20:26.992667 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:26.993205 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:20:26.993205 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:26.993205 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:20:31.411913 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:20:31.412555 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:20:31.412555 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:20:31.413281 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:20:31.451298 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:20:31.451858 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:31.452821 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:20:31.452821 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:31.452821 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:20:31.452821 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:20:31.452821 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:31.452821 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:20:31.452821 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:31.452821 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:20:31.453965 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_fdb_aging_settings failed after 0.040 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:31.453965 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:20:31.453965 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:31.453965 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:20:31.453965 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:20:31.453965 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:20:31.453965 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:20:31.453965 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:31.453965 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:31.453965 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:31.453965 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:20:31.453965 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:31.454545 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:31.454545 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:31.454545 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:20:31.454545 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:31.454545 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:20:31.454545 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:20:31.454545 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:31.454545 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:20:31.454545 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:31.454545 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:20:31.454545 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:20:31.454545 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:31.454545 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:20:31.455175 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:31.455175 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:20:31.455175 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:31.455175 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:20:31.455175 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:20:31.455175 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:20:31.455175 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:20:31.455175 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:20:31.455175 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:31.455175 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:20:31.455175 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:20:31.455175 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:31.455175 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:20:31.455175 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:20:31.455175 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:31.456054 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:31.456054 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:20:31.456054 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:31.456054 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:20:31.456054 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:20:31.456054 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:31.456054 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:31.456054 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:31.456054 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:20:31.456054 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:31.456054 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:31.456054 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:31.456054 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:20:31.456054 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:31.456054 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:20:31.456054 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:20:31.456054 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:31.456054 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:20:31.456054 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:31.456866 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:20:31.456866 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:20:31.456866 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:31.456866 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:20:31.456866 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:31.456866 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:20:31.955579 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:20:31.956201 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:20:31.956402 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:20:31.957404 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b027d63e-13a5-445b-84a0-8ae6223e05d5, routes=[], if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:20:31.957645 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:31.958413 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:20:31.958413 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:31.958413 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:20:31.958413 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:20:31.958413 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:31.958413 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:20:31.958413 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:31.958413 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:20:31.960191 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:31.960191 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:20:31.960191 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:31.960191 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:20:31.960191 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:20:31.960191 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:20:31.960191 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:20:31.960191 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:31.960191 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:31.960191 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:31.960191 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:20:31.960191 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:31.960661 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:31.960661 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:31.960661 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:20:31.960661 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:31.960661 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:20:31.960661 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:20:31.960661 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:31.960661 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:20:31.960661 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:31.960661 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:20:31.960661 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:20:31.960661 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:31.960661 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:20:31.961034 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:31.961034 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:20:31.962661 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:31.962661 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:20:31.962661 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:20:31.962661 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:20:31.962661 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:20:31.962661 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:20:31.962661 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:31.962661 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:20:31.962661 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:20:31.962661 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:31.962661 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:20:31.962661 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:20:31.962661 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:31.962661 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:31.962661 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:20:31.962661 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:31.963141 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:20:31.963141 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:20:31.963141 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:31.963141 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:31.963141 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:31.963141 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:20:31.963141 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:31.963141 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:31.963141 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:31.963141 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:20:31.963141 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:31.963141 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:20:31.963141 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:20:31.963141 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:31.963141 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:20:31.963141 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:31.963141 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:20:31.963141 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:20:31.963141 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:31.963645 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:20:31.963645 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:31.963645 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:20:31.964306 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:20:31.964691 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:20:31.964928 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:20:31.965124 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105458) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 15 10:20:31.965717 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:20:31.966059 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:31.966818 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:20:31.966818 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:31.966818 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:20:31.966818 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:20:31.966818 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:31.966818 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:20:31.966818 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:31.966818 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:20:31.967684 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:31.967684 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:20:31.967684 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:20:31.967684 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:20:31.967684 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:31.967684 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:20:31.967684 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:31.967684 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:20:31.967684 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:31.967684 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:20:31.967684 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:31.967684 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:20:31.967684 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:31.967684 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:20:31.967684 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:20:31.968234 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:31.968234 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:20:31.968234 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:31.968234 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:20:31.968234 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:20:31.968234 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:31.968234 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:20:31.968234 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:31.968234 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:20:31.968624 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:31.968624 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:20:31.968624 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:31.968624 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:20:31.968624 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:20:31.968624 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:20:31.968624 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:20:31.968624 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 15 10:20:31.968624 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:20:31.968624 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:20:31.968624 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:31.968624 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:31.969017 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:31.969017 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:20:31.969017 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:31.969017 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:31.969017 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:31.969017 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:20:31.969017 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:31.969017 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:20:31.969017 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:20:31.969017 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:31.969017 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:20:31.969017 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:31.969017 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:20:31.969017 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:20:31.969428 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:31.969428 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:20:31.969428 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:31.969428 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:20:31.969741 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:31.969741 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:20:31.969741 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:20:31.969741 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:20:31.969741 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:20:31.969741 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:20:31.969741 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:31.969741 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:20:31.969741 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:20:31.969741 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:31.969741 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:20:31.969741 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:20:31.969741 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:31.969741 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:31.969741 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:20:31.969741 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:31.970185 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:20:31.970185 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:20:31.970185 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^ May 15 10:20:31.970185 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:20:31.970185 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:20:31.970185 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:31.970185 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:31.970185 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:31.970185 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:20:31.970185 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:31.970185 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:31.970185 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:31.970185 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:20:31.970185 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:31.970185 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:20:31.970185 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:20:31.970185 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:31.970185 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:20:31.970557 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:31.970557 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:20:31.970557 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:20:31.970557 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:31.970557 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:20:31.970557 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:31.970557 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:20:31.978850 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:20:31.979414 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_ha_failover {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:20:31.979521 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:20:31.979713 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:20:31.980451 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:20:31.980814 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:31.981586 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:20:31.981586 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:31.981586 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:20:31.981586 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:20:31.981586 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:31.981586 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:20:31.981586 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:31.981586 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:20:31.982861 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:31.982861 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:20:31.982861 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:20:31.982861 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:20:31.982861 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:31.982861 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:20:31.982861 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:31.982861 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:20:31.982861 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:31.982861 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:20:31.982861 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:31.982861 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:20:31.982861 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:31.982861 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:20:31.982861 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:20:31.983373 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:31.983373 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:20:31.983373 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:31.983373 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:20:31.983373 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:20:31.983373 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:31.983373 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:20:31.983373 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:31.983373 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:20:31.984311 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:31.984311 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:20:31.984311 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:31.984311 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:20:31.984311 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:20:31.984311 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:20:31.984311 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 15 10:20:31.984311 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:20:31.984311 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:20:31.984311 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:31.984311 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:31.984722 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:31.984722 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:20:31.984722 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:31.984722 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:31.984722 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:31.984722 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:20:31.984722 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:31.984722 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:20:31.984722 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:20:31.984722 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:31.984722 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:20:31.984722 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:31.984722 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:20:31.984722 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:20:31.985167 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:31.985167 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:20:31.985167 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:31.985167 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:20:31.986070 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:31.986070 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:20:31.986070 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:20:31.986070 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:20:31.986070 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:20:31.986070 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:20:31.986070 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:31.986070 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:20:31.986070 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:20:31.986070 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:31.986070 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:20:31.986070 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:20:31.986070 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:31.986070 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:31.986070 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:20:31.986070 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:31.986070 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:20:31.986606 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 15 10:20:31.986606 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:20:31.986606 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:20:31.986606 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:31.986606 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:31.986606 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:31.986606 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:20:31.986606 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:31.986606 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:31.986606 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:31.986606 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:20:31.986606 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:31.986606 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:20:31.986606 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:20:31.986606 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:31.986606 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:20:31.986606 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:31.986606 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:20:31.986606 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:20:31.987505 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:31.987505 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:20:31.987505 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:31.987505 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:20:31.988627 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:20:31.989010 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:20:31.989155 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:20:31.989351 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:20:31.989958 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:20:31.990131 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:20:31.990223 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:31.990720 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:20:31.990720 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:31.990720 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:20:31.990720 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:20:31.990720 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:31.990720 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:20:31.990720 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:31.990720 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:20:31.991643 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:31.991643 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:20:31.991643 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:31.991643 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:20:31.991643 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:20:31.991643 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:20:31.991643 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:20:31.991643 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:31.991643 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:31.991643 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:31.991643 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:20:31.991643 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:31.992085 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:31.992085 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:31.992085 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:20:31.992085 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:31.992085 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:20:31.992085 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:20:31.992085 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:31.992085 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:20:31.992085 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:31.992085 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:20:31.992085 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:20:31.992085 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:31.992085 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:20:31.992565 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:31.992565 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:20:31.992731 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:31.992731 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:20:31.992731 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:20:31.992731 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:20:31.992731 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:20:31.992731 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:20:31.992731 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:31.992731 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:20:31.992731 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:20:31.992731 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:31.992731 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:20:31.992731 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:20:31.992731 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:31.992731 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:31.992731 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:20:31.992731 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:31.993378 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:20:31.993378 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:20:31.993378 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:31.993378 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:31.993378 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:31.993378 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:20:31.993378 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:31.993378 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:31.993378 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:31.993378 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:20:31.993378 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:31.993378 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:20:31.993378 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:20:31.993378 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:31.993378 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:20:31.993378 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:31.993378 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:20:31.993378 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:20:31.993378 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:31.994821 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:20:31.994821 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:31.994821 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:20:36.412508 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:20:36.413263 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:20:36.413446 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:20:36.414299 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:20:36.449410 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:20:36.449896 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:36.450930 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:20:36.450930 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:36.450930 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:20:36.450930 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:20:36.450930 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:36.450930 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:20:36.450930 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:36.450930 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:20:36.452001 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_fdb_aging_settings failed after 0.038 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:36.452001 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:20:36.452001 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:36.452001 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:20:36.452001 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:20:36.452001 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:20:36.452001 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:20:36.452001 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:36.452001 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:36.452001 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:36.452001 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:20:36.452001 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:36.452451 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:36.452451 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:36.452451 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:20:36.452451 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:36.452451 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:20:36.452451 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:20:36.452451 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:36.452451 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:20:36.452451 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:36.452451 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:20:36.452451 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:20:36.452451 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:36.452451 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:20:36.452792 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:36.452792 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:20:36.453198 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:36.453198 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:20:36.453198 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:20:36.453198 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:20:36.453198 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:20:36.453198 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:20:36.453198 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:36.453198 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:20:36.453198 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:20:36.453198 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:36.453198 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:20:36.453198 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:20:36.453198 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:36.453198 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:36.453198 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:20:36.453198 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:36.453811 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:20:36.453811 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:20:36.453811 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:36.453811 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:36.453811 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:36.453811 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:20:36.453811 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:36.453811 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:36.453811 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:36.453811 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:20:36.453811 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:36.453811 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:20:36.453811 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:20:36.453811 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:36.453811 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:20:36.453811 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:36.453811 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:20:36.453811 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:20:36.453811 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:36.454255 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:20:36.454255 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:36.454255 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:20:36.956441 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:20:36.957137 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:20:36.957376 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:20:36.958417 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b027d63e-13a5-445b-84a0-8ae6223e05d5, routes=[], if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:20:36.958723 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:36.959581 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:20:36.959581 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:36.959581 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:20:36.959581 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:20:36.959581 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:36.959581 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:20:36.959581 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:36.959581 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:20:36.961395 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:36.961395 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:20:36.961395 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:36.961395 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:20:36.961395 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:20:36.961395 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:20:36.961395 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:20:36.961395 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:36.961395 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:36.961395 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:36.961395 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:20:36.961395 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:36.961912 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:36.961912 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:36.961912 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:20:36.961912 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:36.961912 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:20:36.961912 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:20:36.961912 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:36.961912 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:20:36.961912 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:36.961912 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:20:36.961912 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:20:36.961912 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:36.961912 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:20:36.962208 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:36.962208 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:20:36.963215 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:36.963215 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:20:36.963215 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:20:36.963215 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:20:36.963215 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:20:36.963215 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:20:36.963215 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:36.963215 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:20:36.963215 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:20:36.963215 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:36.963215 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:20:36.963215 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:20:36.963215 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:36.963215 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:36.963215 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:20:36.963215 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:36.963598 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:20:36.963598 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:20:36.963598 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:36.963598 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:36.963598 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:36.963598 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:20:36.963598 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:36.963598 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:36.963598 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:36.963598 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:20:36.963598 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:36.963598 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:20:36.963598 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:20:36.963598 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:36.963598 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:20:36.963598 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:36.963598 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:20:36.963598 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:20:36.963598 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:36.964068 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:20:36.964068 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:36.964068 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:20:36.964827 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:20:36.965149 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:20:36.965244 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:20:36.965385 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105458) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 15 10:20:36.965851 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:20:36.966066 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:36.966571 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:20:36.966571 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:36.966571 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:20:36.966571 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:20:36.966571 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:36.966571 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:20:36.966571 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:36.966571 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:20:36.967374 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:36.967374 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:20:36.967374 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:20:36.967374 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:20:36.967374 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:36.967374 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:20:36.967374 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:36.967374 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:20:36.967374 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:36.967374 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:20:36.967374 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:36.967374 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:20:36.967374 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:36.967374 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:20:36.967374 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:20:36.968146 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:36.968146 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:20:36.968146 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:36.968146 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:20:36.968146 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:20:36.968146 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:36.968146 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:20:36.968146 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:36.968146 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:20:36.968350 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:36.968350 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:20:36.968350 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:36.968350 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:20:36.968350 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:20:36.968350 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:20:36.968350 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:20:36.968350 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 15 10:20:36.968350 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:20:36.968350 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:20:36.968350 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:36.968350 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:36.968620 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:36.968620 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:20:36.968620 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:36.968620 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:36.968620 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:36.968620 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:20:36.968620 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:36.968620 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:20:36.968620 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:20:36.968620 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:36.968620 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:20:36.968620 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:36.968620 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:20:36.968620 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:20:36.968972 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:36.968972 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:20:36.968972 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:36.968972 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:20:36.969667 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:36.969667 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:20:36.969667 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:20:36.969667 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:20:36.969667 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:20:36.969667 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:20:36.969667 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:36.969667 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:20:36.969667 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:20:36.969667 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:36.969667 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:20:36.969667 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:20:36.969667 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:36.969667 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:36.969667 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:20:36.969667 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:36.970547 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:20:36.970547 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:20:36.970547 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^ May 15 10:20:36.970547 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:20:36.970547 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:20:36.970547 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:36.970547 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:36.970547 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:36.970547 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:20:36.970547 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:36.970547 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:36.970547 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:36.970547 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:20:36.970547 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:36.970547 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:20:36.970547 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:20:36.970547 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:36.970547 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:20:36.971388 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:36.971388 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:20:36.971388 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:20:36.971388 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:36.971388 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:20:36.971388 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:36.971388 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:20:36.979293 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:20:36.979540 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_ha_failover {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:20:36.979648 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:20:36.979802 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:20:36.980349 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:20:36.980656 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:36.981173 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:20:36.981173 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:36.981173 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:20:36.981173 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:20:36.981173 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:36.981173 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:20:36.981173 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:36.981173 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:20:36.981740 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:36.981740 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:20:36.981740 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:20:36.981740 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:20:36.981740 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:36.981740 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:20:36.981740 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:36.981740 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:20:36.981740 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:36.981740 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:20:36.981740 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:36.981740 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:20:36.981740 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:36.981740 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:20:36.981740 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:20:36.982293 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:36.982293 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:20:36.982293 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:36.982293 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:20:36.982293 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:20:36.982293 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:36.982293 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:20:36.982293 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:36.982293 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:20:36.982606 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:36.982606 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:20:36.982606 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:36.982606 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:20:36.982606 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:20:36.982606 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:20:36.982606 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 15 10:20:36.982606 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:20:36.982606 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:20:36.982606 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:36.982606 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:36.983013 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:36.983013 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:20:36.983013 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:36.983013 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:36.983013 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:36.983013 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:20:36.983013 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:36.983013 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:20:36.983013 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:20:36.983013 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:36.983013 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:20:36.983013 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:36.983013 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:20:36.983013 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:20:36.983704 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:36.983704 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:20:36.983704 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:36.983704 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:20:36.983704 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:36.983704 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:20:36.983704 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:20:36.983704 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:20:36.983704 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:20:36.983704 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:20:36.983704 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:36.983704 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:20:36.983704 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:20:36.983704 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:36.984719 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:20:36.984719 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:20:36.984719 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:36.984719 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:36.984719 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:20:36.984719 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:36.984719 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:20:36.984719 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 15 10:20:36.984719 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:20:36.984719 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:20:36.984719 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:36.984719 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:36.984719 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:36.984719 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:20:36.984719 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:36.984719 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:36.984719 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:36.984719 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:20:36.984719 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:36.985989 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:20:36.985989 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:20:36.985989 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:36.985989 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:20:36.985989 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:36.985989 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:20:36.985989 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:20:36.985989 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:36.985989 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:20:36.985989 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:36.985989 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:20:36.989411 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:20:36.989917 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:20:36.990130 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:20:36.990387 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:20:36.991485 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:20:36.992017 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:20:36.992247 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:36.993661 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:20:36.993661 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:36.993661 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:20:36.993661 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:20:36.993661 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:36.993661 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:20:36.993661 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:36.993661 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:20:36.995030 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_mac_aging_settings failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:36.995030 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:20:36.995030 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:36.995030 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:20:36.995030 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:20:36.995030 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:20:36.995030 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:20:36.995030 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:36.995030 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:36.995030 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:36.995030 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:20:36.995030 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:36.995270 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:36.995270 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:36.995270 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:20:36.995270 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:36.995270 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:20:36.995270 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:20:36.995270 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:36.995270 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:20:36.995270 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:36.995270 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:20:36.995270 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:20:36.995270 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:36.995270 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:20:36.995499 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:36.995499 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:20:36.996752 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:36.996752 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:20:36.996752 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:20:36.996752 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:20:36.996752 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:20:36.996752 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:20:36.996752 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:36.996752 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:20:36.996752 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:20:36.996752 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:36.996752 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:20:36.996752 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:20:36.996752 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:36.996752 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:36.996752 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:20:36.996752 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:36.997556 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:20:36.997556 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:20:36.997556 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:36.997556 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:36.997556 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:36.997556 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:20:36.997556 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:36.997556 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:36.997556 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:36.997556 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:20:36.997556 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:36.997556 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:20:36.997556 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:20:36.997556 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:36.997556 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:20:36.997556 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:36.997556 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:20:36.997556 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:20:36.997556 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:36.998309 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:20:36.998309 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:36.998309 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:20:41.413419 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:20:41.413851 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:20:41.413949 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:20:41.414748 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:20:41.453500 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:20:41.453500 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:41.453803 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:20:41.453803 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:41.453803 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:20:41.453803 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:20:41.453803 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:41.453803 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:20:41.453803 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:41.453803 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:20:41.454777 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_fdb_aging_settings failed after 0.040 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:41.454777 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:20:41.454777 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:41.454777 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:20:41.454777 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:20:41.454777 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:20:41.454777 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:20:41.454777 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:41.454777 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:41.454777 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:41.454777 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:20:41.454777 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:41.455093 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:41.455093 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:41.455093 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:20:41.455093 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:41.455093 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:20:41.455093 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:20:41.455093 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:41.455093 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:20:41.455093 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:41.455093 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:20:41.455093 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:20:41.455093 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:41.455093 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:20:41.455464 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:41.455464 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:20:41.455755 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:41.455755 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:20:41.455755 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:20:41.455755 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:20:41.455755 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:20:41.455755 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:20:41.455755 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:41.455755 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:20:41.455755 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:20:41.455755 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:41.455755 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:20:41.455755 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:20:41.455755 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:41.455755 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:41.455755 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:20:41.455755 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:41.456309 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:20:41.456309 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:20:41.456309 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:41.456309 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:41.456309 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:41.456309 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:20:41.456309 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:41.456309 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:41.456309 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:41.456309 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:20:41.456309 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:41.456309 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:20:41.456309 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:20:41.456309 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:41.456309 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:20:41.456309 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:41.456309 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:20:41.456309 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:20:41.456309 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:41.456814 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:20:41.456814 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:41.456814 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:20:41.957144 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:20:41.957619 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:20:41.957770 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:20:41.958706 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b027d63e-13a5-445b-84a0-8ae6223e05d5, routes=[], if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:20:41.958706 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:41.959202 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:20:41.959202 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:41.959202 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:20:41.959202 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:20:41.959202 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:41.959202 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:20:41.959202 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:41.959202 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:20:41.960209 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:41.960209 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:20:41.960209 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:41.960209 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:20:41.960209 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:20:41.960209 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:20:41.960209 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:20:41.960209 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:41.960209 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:41.960209 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:41.960209 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:20:41.960209 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:41.960574 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:41.960574 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:41.960574 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:20:41.960574 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:41.960574 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:20:41.960574 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:20:41.960574 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:41.960574 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:20:41.960574 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:41.960574 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:20:41.960574 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:20:41.960574 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:41.960574 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:20:41.960915 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:41.960915 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:20:41.961316 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:41.961316 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:20:41.961316 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:20:41.961316 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:20:41.961316 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:20:41.961316 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:20:41.961316 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:41.961316 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:20:41.961316 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:20:41.961316 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:41.961316 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:20:41.961316 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:20:41.961316 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:41.961316 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:41.961316 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:20:41.961316 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:41.961762 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:20:41.961762 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:20:41.961762 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:41.961762 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:41.961762 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:41.961762 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:20:41.961762 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:41.961762 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:41.961762 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:41.961762 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:20:41.961762 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:41.961762 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:20:41.961762 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:20:41.961762 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:41.961762 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:20:41.961762 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:41.961762 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:20:41.961762 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:20:41.961762 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:41.962191 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:20:41.962191 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:41.962191 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:20:41.965362 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:20:41.965638 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:20:41.965797 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:20:41.965933 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105458) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 15 10:20:41.966753 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:20:41.966753 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:41.966882 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:20:41.966882 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:41.966882 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:20:41.966882 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:20:41.966882 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:41.966882 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:20:41.966882 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:41.966882 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:20:41.967635 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:41.967635 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:20:41.967635 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:20:41.967635 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:20:41.967635 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:41.967635 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:20:41.967635 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:41.967635 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:20:41.967635 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:41.967635 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:20:41.967635 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:41.967635 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:20:41.967635 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:41.967635 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:20:41.967635 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:20:41.968110 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:41.968110 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:20:41.968110 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:41.968110 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:20:41.968110 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:20:41.968110 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:41.968110 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:20:41.968110 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:41.968110 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:20:41.968548 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:41.968548 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:20:41.968548 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:41.968548 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:20:41.968548 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:20:41.968548 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:20:41.968548 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:20:41.968548 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 15 10:20:41.968548 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:20:41.968548 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:20:41.968548 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:41.968548 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:41.968961 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:41.968961 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:20:41.968961 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:41.968961 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:41.968961 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:41.968961 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:20:41.968961 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:41.968961 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:20:41.968961 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:20:41.968961 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:41.968961 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:20:41.968961 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:41.968961 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:20:41.968961 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:20:41.969347 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:41.969347 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:20:41.969347 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:41.969347 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:20:41.969615 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:41.969615 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:20:41.969615 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:20:41.969615 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:20:41.969615 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:20:41.969615 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:20:41.969615 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:41.969615 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:20:41.969615 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:20:41.969615 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:41.969615 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:20:41.969615 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:20:41.969615 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:41.969615 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:41.969615 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:20:41.969615 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:41.970091 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:20:41.970091 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:20:41.970091 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^ May 15 10:20:41.970091 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:20:41.970091 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:20:41.970091 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:41.970091 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:41.970091 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:41.970091 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:20:41.970091 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:41.970091 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:41.970091 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:41.970091 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:20:41.970091 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:41.970091 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:20:41.970091 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:20:41.970091 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:41.970091 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:20:41.970551 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:41.970551 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:20:41.970551 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:20:41.970551 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:41.970551 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:20:41.970551 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:41.970551 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:20:41.979780 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:20:41.980219 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_ha_failover {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:20:41.980327 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:20:41.980525 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:20:41.981536 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:20:41.981536 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:41.981742 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:20:41.981742 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:41.981742 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:20:41.981742 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:20:41.981742 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:41.981742 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:20:41.981742 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:41.981742 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:20:41.982314 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:41.982314 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:20:41.982314 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:20:41.982314 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:20:41.982314 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:41.982314 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:20:41.982314 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:41.982314 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:20:41.982314 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:41.982314 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:20:41.982314 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:41.982314 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:20:41.982314 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:41.982314 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:20:41.982314 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:20:41.982740 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:41.982740 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:20:41.982740 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:41.982740 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:20:41.982740 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:20:41.982740 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:41.982740 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:20:41.982740 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:41.982740 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:20:41.983204 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:41.983204 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:20:41.983204 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:41.983204 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:20:41.983204 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:20:41.983204 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:20:41.983204 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 15 10:20:41.983204 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:20:41.983204 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:20:41.983204 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:41.983204 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:41.983512 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:41.983512 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:20:41.983512 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:41.983512 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:41.983512 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:41.983512 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:20:41.983512 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:41.983512 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:20:41.983512 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:20:41.983512 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:41.983512 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:20:41.983512 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:41.983512 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:20:41.983512 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:20:41.983916 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:41.983916 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:20:41.983916 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:41.983916 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:20:41.984032 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:41.984032 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:20:41.984032 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:20:41.984032 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:20:41.984032 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:20:41.984032 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:20:41.984032 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:41.984032 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:20:41.984032 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:20:41.984032 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:41.984032 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:20:41.984032 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:20:41.984032 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:41.984032 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:41.984032 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:20:41.984032 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:41.984032 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:20:41.984531 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 15 10:20:41.984531 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:20:41.984531 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:20:41.984531 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:41.984531 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:41.984531 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:41.984531 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:20:41.984531 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:41.984531 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:41.984531 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:41.984531 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:20:41.984531 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:41.984531 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:20:41.984531 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:20:41.984531 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:41.984531 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:20:41.984531 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:41.984531 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:20:41.984531 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:20:41.985052 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:41.985052 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:20:41.985052 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:41.985052 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:20:41.989918 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:20:41.990375 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:20:41.990550 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:20:41.990825 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:20:41.991148 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:20:41.991482 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:20:41.991718 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:41.992354 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:20:41.992354 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:41.992354 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:20:41.992354 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:20:41.992354 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:41.992354 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:20:41.992354 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:41.992354 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:20:41.993695 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:41.993695 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:20:41.993695 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:41.993695 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:20:41.993695 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:20:41.993695 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:20:41.993695 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:20:41.993695 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:41.993695 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:41.993695 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:41.993695 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:20:41.993695 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:41.994084 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:41.994084 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:41.994084 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:20:41.994084 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:41.994084 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:20:41.994084 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:20:41.994084 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:41.994084 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:20:41.994084 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:41.994084 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:20:41.994084 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:20:41.994084 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:41.994084 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:20:41.994558 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:41.994558 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:20:41.995169 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:41.995169 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:20:41.995169 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:20:41.995169 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:20:41.995169 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:20:41.995169 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:20:41.995169 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:41.995169 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:20:41.995169 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:20:41.995169 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:41.995169 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:20:41.995169 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:20:41.995169 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:41.995169 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:41.995169 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:20:41.995169 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:41.995641 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:20:41.995641 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:20:41.995641 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:41.995641 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:41.995641 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:41.995641 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:20:41.995641 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:41.995641 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:41.995641 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:41.995641 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:20:41.995641 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:41.995641 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:20:41.995641 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:20:41.995641 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:41.995641 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:20:41.995641 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:41.995641 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:20:41.995641 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:20:41.995641 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:41.996376 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:20:41.996376 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:41.996376 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:20:46.414157 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:20:46.414955 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:20:46.415022 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:20:46.416111 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:20:46.449757 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:20:46.449990 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:46.450508 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:20:46.450508 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:46.450508 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:20:46.450508 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:20:46.450508 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:46.450508 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:20:46.450508 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:46.450508 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:20:46.451200 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_fdb_aging_settings failed after 0.035 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:46.451200 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:20:46.451200 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:46.451200 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:20:46.451200 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:20:46.451200 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:20:46.451200 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:20:46.451200 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:46.451200 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:46.451200 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:46.451200 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:20:46.451200 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:46.451582 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:46.451582 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:46.451582 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:20:46.451582 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:46.451582 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:20:46.451582 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:20:46.451582 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:46.451582 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:20:46.451582 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:46.451582 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:20:46.451582 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:20:46.451582 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:46.451582 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:20:46.452094 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:46.452094 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:20:46.452094 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:46.452094 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:20:46.452094 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:20:46.452094 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:20:46.452094 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:20:46.452094 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:20:46.452094 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:46.452094 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:20:46.452094 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:20:46.452094 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:46.452094 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:20:46.452094 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:20:46.452094 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:46.452598 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:46.452598 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:20:46.452598 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:46.452598 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:20:46.452598 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:20:46.452598 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:46.452598 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:46.452598 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:46.452598 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:20:46.452598 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:46.452598 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:46.452598 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:46.452598 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:20:46.452598 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:46.452598 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:20:46.452598 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:20:46.452598 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:46.452598 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:20:46.452598 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:46.453286 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:20:46.453286 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:20:46.453286 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:46.453286 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:20:46.453286 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:46.453286 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:20:46.957944 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:20:46.958493 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:20:46.958587 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:20:46.959405 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b027d63e-13a5-445b-84a0-8ae6223e05d5, routes=[], if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:20:46.959645 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:46.960347 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:20:46.960347 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:46.960347 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:20:46.960347 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:20:46.960347 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:46.960347 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:20:46.960347 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:46.960347 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:20:46.961275 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:46.961275 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:20:46.961275 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:46.961275 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:20:46.961275 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:20:46.961275 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:20:46.961275 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:20:46.961275 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:46.961275 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:46.961275 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:46.961275 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:20:46.961275 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:46.961897 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:46.961897 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:46.961897 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:20:46.961897 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:46.961897 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:20:46.961897 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:20:46.961897 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:46.961897 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:20:46.961897 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:46.961897 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:20:46.961897 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:20:46.961897 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:46.961897 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:20:46.962462 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:46.962462 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:20:46.962462 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:46.962462 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:20:46.962462 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:20:46.962462 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:20:46.962462 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:20:46.962462 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:20:46.962462 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:46.962462 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:20:46.962462 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:20:46.962462 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:46.962462 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:20:46.962462 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:20:46.962462 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:46.963210 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:46.963210 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:20:46.963210 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:46.963210 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:20:46.963210 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:20:46.963210 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:46.963210 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:46.963210 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:46.963210 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:20:46.963210 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:46.963210 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:46.963210 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:46.963210 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:20:46.963210 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:46.963210 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:20:46.963210 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:20:46.963210 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:46.963210 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:20:46.963210 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:46.964311 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:20:46.964311 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:20:46.964311 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:46.964311 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:20:46.964311 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:46.964311 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:20:46.965852 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:20:46.966190 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:20:46.966315 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:20:46.966444 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105458) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 15 10:20:46.966905 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:20:46.967143 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:46.967692 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:20:46.967692 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:46.967692 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:20:46.967692 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:20:46.967692 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:46.967692 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:20:46.967692 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:46.967692 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:20:46.968603 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:46.968603 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:20:46.968603 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:20:46.968603 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:20:46.968603 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:46.968603 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:20:46.968603 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:46.968603 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:20:46.968603 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:46.968603 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:20:46.968603 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:46.968603 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:20:46.968603 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:46.968603 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:20:46.968603 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:20:46.969061 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:46.969061 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:20:46.969061 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:46.969061 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:20:46.969061 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:20:46.969061 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:46.969061 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:20:46.969061 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:46.969061 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:20:46.969485 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:46.969485 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:20:46.969485 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:46.969485 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:20:46.969485 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:20:46.969485 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:20:46.969485 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:20:46.969485 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 15 10:20:46.969485 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:20:46.969485 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:20:46.969485 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:46.969485 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:46.969841 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:46.969841 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:20:46.969841 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:46.969841 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:46.969841 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:46.969841 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:20:46.969841 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:46.969841 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:20:46.969841 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:20:46.969841 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:46.969841 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:20:46.969841 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:46.969841 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:20:46.969841 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:20:46.970212 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:46.970212 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:20:46.970212 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:46.970212 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:20:46.970610 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:46.970610 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:20:46.970610 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:20:46.970610 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:20:46.970610 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:20:46.970610 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:20:46.970610 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:46.970610 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:20:46.970610 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:20:46.970610 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:46.970610 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:20:46.970610 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:20:46.970610 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:46.970610 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:46.970610 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:20:46.970610 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:46.971028 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:20:46.971028 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:20:46.971028 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^ May 15 10:20:46.971028 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:20:46.971028 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:20:46.971028 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:46.971028 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:46.971028 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:46.971028 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:20:46.971028 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:46.971028 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:46.971028 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:46.971028 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:20:46.971028 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:46.971028 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:20:46.971028 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:20:46.971028 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:46.971028 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:20:46.971644 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:46.971644 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:20:46.971644 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:20:46.971644 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:46.971644 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:20:46.971644 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:46.971644 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:20:46.980548 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:20:46.980946 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_ha_failover {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:20:46.981064 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:20:46.981226 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:20:46.981846 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:20:46.982245 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:46.983146 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:20:46.983146 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:46.983146 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:20:46.983146 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:20:46.983146 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:46.983146 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:20:46.983146 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:46.983146 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:20:46.984666 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:46.984666 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:20:46.984666 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:20:46.984666 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:20:46.984666 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:46.984666 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:20:46.984666 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:46.984666 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:20:46.984666 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:46.984666 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:20:46.984666 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:46.984666 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:20:46.984666 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:46.984666 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:20:46.984666 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:20:46.985118 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:46.985118 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:20:46.985118 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:46.985118 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:20:46.985118 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:20:46.985118 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:46.985118 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:20:46.985118 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:46.985118 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:20:46.986349 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_ha_failover failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:46.986349 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:20:46.986349 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:46.986349 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:20:46.986349 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:20:46.986349 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:20:46.986349 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 15 10:20:46.986349 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:20:46.986349 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:20:46.986349 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:46.986349 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:46.986820 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:46.986820 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:20:46.986820 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:46.986820 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:46.986820 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:46.986820 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:20:46.986820 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:46.986820 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:20:46.986820 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:20:46.986820 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:46.986820 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:20:46.986820 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:46.986820 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:20:46.986820 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:20:46.987214 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:46.987214 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:20:46.987214 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:46.987214 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:20:46.988473 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:46.988473 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:20:46.988473 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:20:46.988473 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:20:46.988473 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:20:46.988473 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:20:46.988473 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:46.988473 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:20:46.988473 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:20:46.988473 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:46.988473 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:20:46.988473 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:20:46.988473 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:46.988473 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:46.988473 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:20:46.988473 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:46.988473 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:20:46.988978 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 15 10:20:46.988978 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:20:46.988978 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:20:46.988978 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:46.988978 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:46.988978 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:46.988978 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:20:46.988978 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:46.988978 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:46.988978 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:46.988978 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:20:46.988978 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:46.988978 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:20:46.988978 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:20:46.988978 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:46.988978 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:20:46.988978 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:46.988978 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:20:46.988978 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:20:46.989494 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:46.989494 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:20:46.989494 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:46.989494 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:20:46.990405 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:20:46.990721 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:20:46.990875 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:20:46.991075 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:20:46.991695 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:20:46.991961 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:20:46.992154 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:46.992907 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:20:46.992907 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:46.992907 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:20:46.992907 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:20:46.992907 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:46.992907 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:20:46.992907 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:46.992907 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:20:46.993789 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:46.993789 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:20:46.993789 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:46.993789 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:20:46.993789 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:20:46.993789 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:20:46.993789 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:20:46.993789 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:46.993789 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:46.993789 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:46.993789 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:20:46.993789 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:46.994159 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:46.994159 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:46.994159 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:20:46.994159 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:46.994159 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:20:46.994159 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:20:46.994159 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:46.994159 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:20:46.994159 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:46.994159 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:20:46.994159 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:20:46.994159 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:46.994159 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:20:46.994525 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:46.994525 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:20:46.994768 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:46.994768 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:20:46.994768 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:20:46.994768 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:20:46.994768 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:20:46.994768 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:20:46.994768 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:46.994768 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:20:46.994768 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:20:46.994768 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:46.994768 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:20:46.994768 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:20:46.994768 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:46.994768 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:46.994768 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:20:46.994768 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:46.995246 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:20:46.995246 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:20:46.995246 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:46.995246 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:46.995246 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:46.995246 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:20:46.995246 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:46.995246 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:46.995246 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:46.995246 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:20:46.995246 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:46.995246 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:20:46.995246 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:20:46.995246 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:46.995246 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:20:46.995246 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:46.995246 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:20:46.995246 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:20:46.995246 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:46.995804 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:20:46.995804 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:46.995804 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:20:51.414771 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:20:51.415738 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:20:51.415738 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:20:51.416921 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:20:51.418609 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG dbcounter [-] [105458] Writing DB stats neutron:SELECT=260 {{(pid=105458) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} May 15 10:20:51.451854 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:20:51.452033 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:51.452493 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:20:51.452493 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:51.452493 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:20:51.452493 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:20:51.452493 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:51.452493 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:20:51.452493 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:51.452493 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:20:51.453266 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_fdb_aging_settings failed after 0.037 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:51.453266 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:20:51.453266 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:51.453266 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:20:51.453266 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:20:51.453266 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:20:51.453266 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:20:51.453266 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:51.453266 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:51.453266 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:51.453266 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:20:51.453266 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:51.453880 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:51.453880 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:51.453880 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:20:51.453880 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:51.453880 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:20:51.453880 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:20:51.453880 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:51.453880 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:20:51.453880 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:51.453880 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:20:51.453880 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:20:51.453880 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:51.453880 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:20:51.454384 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:51.454384 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:20:51.454384 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:51.454384 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:20:51.454384 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:20:51.454384 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:20:51.454384 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:20:51.454384 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:20:51.454384 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:51.454384 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:20:51.454384 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:20:51.454384 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:51.454384 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:20:51.454384 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:20:51.454384 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:51.455072 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:51.455072 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:20:51.455072 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:51.455072 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:20:51.455072 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:20:51.455072 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:51.455072 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:51.455072 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:51.455072 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:20:51.455072 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:51.455072 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:51.455072 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:51.455072 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:20:51.455072 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:51.455072 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:20:51.455072 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:20:51.455072 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:51.455072 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:20:51.455072 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:51.455857 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:20:51.455857 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:20:51.455857 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:51.455857 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:20:51.455857 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:51.455857 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:20:51.958593 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:20:51.959090 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:20:51.959427 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:20:51.960854 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b027d63e-13a5-445b-84a0-8ae6223e05d5, routes=[], if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:20:51.961305 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:51.962416 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:20:51.962416 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:51.962416 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:20:51.962416 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:20:51.962416 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:51.962416 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:20:51.962416 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:51.962416 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:20:51.964513 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:51.964513 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:20:51.964513 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:51.964513 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:20:51.964513 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:20:51.964513 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:20:51.964513 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:20:51.964513 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:51.964513 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:51.964513 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:51.964513 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:20:51.964513 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:51.965378 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:51.965378 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:51.965378 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:20:51.965378 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:51.965378 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:20:51.965378 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:20:51.965378 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:51.965378 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:20:51.965378 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:51.965378 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:20:51.965378 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:20:51.965378 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:51.965378 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:20:51.966249 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:51.966249 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:20:51.966958 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:51.966958 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:20:51.966958 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:20:51.966958 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:20:51.966958 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:20:51.966958 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:20:51.966958 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:51.966958 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:20:51.966958 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:20:51.966958 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:51.966958 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:20:51.966958 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:20:51.966958 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:51.966958 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:51.966958 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:20:51.966958 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:51.967821 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:20:51.967821 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:20:51.967821 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:51.967821 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:51.967821 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:51.967821 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:20:51.967821 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:51.967821 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:51.967821 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:51.967821 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:20:51.967821 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:51.967821 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:20:51.967821 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:20:51.967821 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:51.967821 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:20:51.967821 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:51.967821 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:20:51.967821 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:20:51.967821 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:51.968430 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:20:51.968430 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:51.968430 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:20:51.968430 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:20:51.968430 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:20:51.968430 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:20:51.968430 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105458) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 15 10:20:51.968658 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:20:51.969034 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:51.969923 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:20:51.969923 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:51.969923 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:20:51.969923 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:20:51.969923 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:51.969923 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:20:51.969923 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:51.969923 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:20:51.970871 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:51.970871 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:20:51.970871 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:20:51.970871 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:20:51.970871 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:51.970871 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:20:51.970871 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:51.970871 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:20:51.970871 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:51.970871 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:20:51.970871 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:51.970871 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:20:51.970871 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:51.970871 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:20:51.970871 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:20:51.971517 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:51.971517 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:20:51.971517 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:51.971517 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:20:51.971517 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:20:51.971517 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:51.971517 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:20:51.971517 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:51.971517 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:20:51.972081 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:51.972081 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:20:51.972081 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:51.972081 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:20:51.972081 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:20:51.972081 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:20:51.972081 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:20:51.972081 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 15 10:20:51.972081 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:20:51.972081 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:20:51.972081 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:51.972081 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:51.972553 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:51.972553 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:20:51.972553 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:51.972553 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:51.972553 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:51.972553 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:20:51.972553 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:51.972553 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:20:51.972553 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:20:51.972553 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:51.972553 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:20:51.972553 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:51.972553 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:20:51.972553 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:20:51.973142 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:51.973142 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:20:51.973142 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:51.973142 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:20:51.973487 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:51.973487 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:20:51.973487 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:20:51.973487 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:20:51.973487 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:20:51.973487 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:20:51.973487 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:51.973487 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:20:51.973487 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:20:51.973487 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:51.973487 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:20:51.973487 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:20:51.973487 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:51.973487 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:51.973487 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:20:51.973487 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:51.973975 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:20:51.973975 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:20:51.973975 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^ May 15 10:20:51.973975 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:20:51.973975 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:20:51.973975 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:51.973975 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:51.973975 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:51.973975 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:20:51.973975 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:51.973975 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:51.973975 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:51.973975 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:20:51.973975 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:51.973975 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:20:51.973975 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:20:51.973975 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:51.973975 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:20:51.974425 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:51.974425 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:20:51.974425 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:20:51.974425 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:51.974425 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:20:51.974425 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:51.974425 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:20:51.980986 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:20:51.981333 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_ha_failover {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:20:51.981525 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:20:51.981781 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:20:51.982465 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:20:51.982829 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:51.983732 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:20:51.983732 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:51.983732 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:20:51.983732 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:20:51.983732 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:51.983732 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:20:51.983732 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:51.983732 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:20:51.985259 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:51.985259 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:20:51.985259 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:20:51.985259 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:20:51.985259 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:51.985259 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:20:51.985259 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:51.985259 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:20:51.985259 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:51.985259 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:20:51.985259 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:51.985259 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:20:51.985259 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:51.985259 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:20:51.985259 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:20:51.986248 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:51.986248 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:20:51.986248 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:51.986248 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:20:51.986248 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:20:51.986248 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:51.986248 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:20:51.986248 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:51.986248 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:20:51.987241 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_ha_failover failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:51.987241 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:20:51.987241 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:51.987241 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:20:51.987241 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:20:51.987241 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:20:51.987241 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 15 10:20:51.987241 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:20:51.987241 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:20:51.987241 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:51.987241 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:51.988260 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:51.988260 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:20:51.988260 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:51.988260 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:51.988260 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:51.988260 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:20:51.988260 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:51.988260 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:20:51.988260 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:20:51.988260 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:51.988260 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:20:51.988260 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:51.988260 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:20:51.988260 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:20:51.989322 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:51.989322 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:20:51.989322 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:51.989322 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:20:51.989885 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:51.989885 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:20:51.989885 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:20:51.989885 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:20:51.989885 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:20:51.989885 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:20:51.989885 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:51.989885 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:20:51.989885 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:20:51.989885 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:51.989885 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:20:51.989885 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:20:51.989885 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:51.989885 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:51.989885 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:20:51.989885 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:51.989885 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:20:51.990406 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 15 10:20:51.990406 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:20:51.990406 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:20:51.990406 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:51.990406 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:51.990406 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:51.990406 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:20:51.990406 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:51.990406 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:51.990406 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:51.990406 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:20:51.990406 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:51.990406 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:20:51.990406 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:20:51.990406 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:51.990406 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:20:51.990406 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:51.990406 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:20:51.990406 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:20:51.991003 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:51.991003 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:20:51.991003 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:51.991003 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:20:51.991003 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:20:51.991244 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:20:51.991391 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:20:51.991596 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:20:51.992173 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:20:51.992550 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:20:51.992776 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:51.993600 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:20:51.993600 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:51.993600 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:20:51.993600 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:20:51.993600 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:51.993600 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:20:51.993600 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:51.993600 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:20:51.994647 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:51.994647 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:20:51.994647 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:51.994647 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:20:51.994647 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:20:51.994647 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:20:51.994647 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:20:51.994647 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:51.994647 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:51.994647 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:51.994647 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:20:51.994647 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:51.995180 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:51.995180 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:51.995180 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:20:51.995180 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:51.995180 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:20:51.995180 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:20:51.995180 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:51.995180 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:20:51.995180 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:51.995180 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:20:51.995180 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:20:51.995180 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:51.995180 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:20:51.995756 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:51.995756 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:20:51.996040 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:51.996040 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:20:51.996040 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:20:51.996040 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:20:51.996040 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:20:51.996040 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:20:51.996040 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:51.996040 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:20:51.996040 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:20:51.996040 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:51.996040 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:20:51.996040 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:20:51.996040 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:51.996040 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:51.996040 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:20:51.996040 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:51.996540 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:20:51.996540 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:20:51.996540 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:51.996540 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:51.996540 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:51.996540 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:20:51.996540 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:51.996540 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:51.996540 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:51.996540 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:20:51.996540 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:51.996540 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:20:51.996540 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:20:51.996540 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:51.996540 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:20:51.996540 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:51.996540 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:20:51.996540 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:20:51.996540 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:51.997070 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:20:51.997070 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:51.997070 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:20:56.415505 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:20:56.416504 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:20:56.416504 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:20:56.417602 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:20:56.452667 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:20:56.453174 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:56.454173 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:20:56.454173 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:56.454173 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:20:56.454173 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:20:56.454173 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:56.454173 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:20:56.454173 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:56.454173 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:20:56.454950 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_fdb_aging_settings failed after 0.038 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:56.454950 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:20:56.454950 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:56.454950 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:20:56.454950 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:20:56.454950 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:20:56.454950 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:20:56.454950 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:56.454950 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:56.454950 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:56.454950 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:20:56.454950 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:56.455562 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:56.455562 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:56.455562 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:20:56.455562 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:56.455562 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:20:56.455562 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:20:56.455562 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:56.455562 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:20:56.455562 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:56.455562 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:20:56.455562 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:20:56.455562 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:56.455562 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:20:56.456240 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:56.456240 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:20:56.456240 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:56.456240 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:20:56.456240 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:20:56.456240 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:20:56.456240 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:20:56.456240 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:20:56.456240 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:56.456240 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:20:56.456240 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:20:56.456240 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:56.456240 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:20:56.456240 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:20:56.456240 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:56.457176 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:56.457176 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:20:56.457176 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:56.457176 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:20:56.457176 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:20:56.457176 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:56.457176 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:56.457176 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:56.457176 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:20:56.457176 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:56.457176 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:56.457176 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:56.457176 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:20:56.457176 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:56.457176 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:20:56.457176 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:20:56.457176 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:56.457176 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:20:56.457176 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:56.458493 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:20:56.458493 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:20:56.458493 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:56.458493 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:20:56.458493 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:56.458493 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:20:56.959470 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:20:56.960321 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:20:56.960623 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:20:56.962042 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b027d63e-13a5-445b-84a0-8ae6223e05d5, routes=[], if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:20:56.962347 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:56.963402 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:20:56.963402 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:56.963402 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:20:56.963402 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:20:56.963402 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:56.963402 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:20:56.963402 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:56.963402 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:20:56.965504 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:56.965504 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:20:56.965504 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:56.965504 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:20:56.965504 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:20:56.965504 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:20:56.965504 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:20:56.965504 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:56.965504 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:56.965504 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:56.965504 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:20:56.965504 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:56.966636 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:56.966636 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:56.966636 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:20:56.966636 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:56.966636 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:20:56.966636 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:20:56.966636 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:56.966636 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:20:56.966636 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:56.966636 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:20:56.966636 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:20:56.966636 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:56.966636 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:20:56.967585 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:56.967585 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:20:56.968374 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:56.968374 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:20:56.968374 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:20:56.968374 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:20:56.968374 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:20:56.968374 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:20:56.968374 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:56.968374 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:20:56.968374 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:20:56.968374 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:56.968374 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:20:56.968374 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:20:56.968374 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:56.968374 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:56.968374 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:20:56.968374 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:56.969261 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:20:56.969261 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:20:56.969261 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:56.969261 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:56.969261 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:56.969261 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:20:56.969261 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:56.969261 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:56.969261 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:56.969261 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:20:56.969261 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:56.969261 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:20:56.969261 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:20:56.969261 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:56.969261 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:20:56.969261 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:56.969261 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:20:56.969261 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:20:56.969261 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:56.970654 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:20:56.970654 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:56.970654 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:20:56.970654 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:20:56.970654 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:20:56.970654 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:20:56.970654 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105458) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 15 10:20:56.970911 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:20:56.971253 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:56.972086 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:20:56.972086 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:56.972086 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:20:56.972086 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:20:56.972086 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:56.972086 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:20:56.972086 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:56.972086 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:20:56.973338 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:56.973338 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:20:56.973338 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:20:56.973338 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:20:56.973338 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:56.973338 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:20:56.973338 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:56.973338 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:20:56.973338 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:56.973338 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:20:56.973338 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:56.973338 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:20:56.973338 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:56.973338 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:20:56.973338 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:20:56.973642 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:56.973642 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:20:56.973642 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:56.973642 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:20:56.973642 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:20:56.973642 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:56.973642 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:20:56.973642 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:56.973642 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:20:56.974853 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:56.974853 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:20:56.974853 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:56.974853 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:20:56.974853 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:20:56.974853 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:20:56.974853 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:20:56.974853 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 15 10:20:56.974853 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:20:56.974853 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:20:56.974853 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:56.974853 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:56.975101 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:56.975101 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:20:56.975101 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:56.975101 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:56.975101 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:56.975101 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:20:56.975101 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:56.975101 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:20:56.975101 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:20:56.975101 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:56.975101 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:20:56.975101 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:56.975101 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:20:56.975101 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:20:56.975387 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:56.975387 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:20:56.975387 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:56.975387 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:20:56.976516 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:56.976516 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:20:56.976516 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:20:56.976516 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:20:56.976516 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:20:56.976516 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:20:56.976516 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:56.976516 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:20:56.976516 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:20:56.976516 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:56.976516 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:20:56.976516 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:20:56.976516 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:56.976516 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:56.976516 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:20:56.976516 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:56.976856 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:20:56.976856 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:20:56.976856 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^ May 15 10:20:56.976856 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:20:56.976856 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:20:56.976856 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:56.976856 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:56.976856 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:56.976856 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:20:56.976856 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:56.976856 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:56.976856 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:56.976856 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:20:56.976856 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:56.976856 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:20:56.976856 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:20:56.976856 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:56.976856 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:20:56.977180 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:56.977180 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:20:56.977180 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:20:56.977180 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:56.977180 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:20:56.977180 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:56.977180 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:20:56.981418 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:20:56.981784 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_ha_failover {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:20:56.981904 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:20:56.982038 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:20:56.982862 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:20:56.983289 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:56.984100 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:20:56.984100 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:56.984100 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:20:56.984100 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:20:56.984100 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:56.984100 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:20:56.984100 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:56.984100 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:20:56.985062 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:56.985062 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:20:56.985062 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:20:56.985062 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:20:56.985062 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:56.985062 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:20:56.985062 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:56.985062 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:20:56.985062 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:56.985062 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:20:56.985062 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:56.985062 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:20:56.985062 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:56.985062 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:20:56.985062 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:20:56.985794 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:56.985794 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:20:56.985794 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:56.985794 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:20:56.985794 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:20:56.985794 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:56.985794 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:20:56.985794 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:56.985794 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:20:56.986189 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:56.986189 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:20:56.986189 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:56.986189 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:20:56.986189 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:20:56.986189 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:20:56.986189 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 15 10:20:56.986189 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:20:56.986189 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:20:56.986189 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:56.986189 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:56.986760 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:56.986760 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:20:56.986760 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:56.986760 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:56.986760 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:56.986760 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:20:56.986760 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:56.986760 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:20:56.986760 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:20:56.986760 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:56.986760 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:20:56.986760 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:56.986760 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:20:56.986760 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:20:56.987543 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:56.987543 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:20:56.987543 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:56.987543 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:20:56.987543 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:56.987543 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:20:56.987543 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:20:56.987543 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:20:56.987543 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:20:56.987543 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:20:56.987543 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:56.987543 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:20:56.987543 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:20:56.987543 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:56.988202 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:20:56.988202 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:20:56.988202 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:56.988202 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:56.988202 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:20:56.988202 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:56.988202 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:20:56.988202 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 15 10:20:56.988202 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:20:56.988202 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:20:56.988202 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:56.988202 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:56.988202 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:56.988202 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:20:56.988202 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:56.988202 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:56.988202 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:56.988202 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:20:56.988202 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:56.989087 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:20:56.989087 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:20:56.989087 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:56.989087 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:20:56.989087 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:56.989087 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:20:56.989087 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:20:56.989087 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:56.989087 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:20:56.989087 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:56.989087 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:20:56.991309 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:20:56.991588 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:20:56.991836 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:20:56.991995 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:20:56.992519 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:20:56.992989 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:20:56.993250 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:56.994069 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:20:56.994069 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:56.994069 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:20:56.994069 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:20:56.994069 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:56.994069 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:20:56.994069 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:56.994069 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:20:56.995053 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:56.995053 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:20:56.995053 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:56.995053 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:20:56.995053 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:20:56.995053 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:20:56.995053 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:20:56.995053 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:56.995053 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:56.995053 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:56.995053 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:20:56.995053 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:56.995375 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:20:56.995375 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:56.995375 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:20:56.995375 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:56.995375 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:20:56.995375 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:20:56.995375 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:56.995375 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:20:56.995375 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:56.995375 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:20:56.995375 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:20:56.995375 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:56.995375 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:20:56.995708 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:56.995708 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:20:56.996014 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:56.996014 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:20:56.996014 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:20:56.996014 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:20:56.996014 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:20:56.996014 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:20:56.996014 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:56.996014 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:20:56.996014 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:20:56.996014 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:56.996014 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:20:56.996014 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:20:56.996014 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:20:56.996014 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:20:56.996014 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:20:56.996014 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:20:56.996410 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:20:56.996410 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:20:56.996410 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:56.996410 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:56.996410 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:20:56.996410 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:20:56.996410 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:20:56.996410 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:20:56.996410 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:20:56.996410 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:20:56.996410 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:20:56.996410 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:20:56.996410 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:20:56.996410 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:20:56.996410 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:20:56.996410 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:20:56.996410 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:20:56.996410 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:20:56.996410 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:20:56.996950 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:20:56.996950 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:20:56.996950 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:21:01.416305 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:21:01.417613 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:21:01.417613 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:21:01.418483 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:21:01.450149 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:21:01.450484 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:01.451239 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:21:01.451239 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:01.451239 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:21:01.451239 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:21:01.451239 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:01.451239 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:21:01.451239 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:01.451239 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:21:01.451974 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_fdb_aging_settings failed after 0.034 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:01.451974 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:21:01.451974 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:01.451974 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:21:01.451974 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:21:01.451974 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:21:01.451974 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:21:01.451974 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:01.451974 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:01.451974 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:01.451974 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:21:01.451974 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:01.452568 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:01.452568 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:01.452568 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:21:01.452568 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:01.452568 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:21:01.452568 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:21:01.452568 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:01.452568 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:21:01.452568 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:01.452568 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:21:01.452568 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:21:01.452568 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:01.452568 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:21:01.453150 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:01.453150 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:21:01.453150 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:01.453150 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:21:01.453150 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:21:01.453150 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:21:01.453150 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:21:01.453150 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:21:01.453150 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:01.453150 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:21:01.453150 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:21:01.453150 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:01.453150 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:21:01.453150 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:21:01.453150 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:01.454324 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:01.454324 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:21:01.454324 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:01.454324 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:21:01.454324 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:21:01.454324 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:01.454324 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:01.454324 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:01.454324 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:21:01.454324 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:01.454324 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:01.454324 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:01.454324 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:21:01.454324 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:01.454324 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:21:01.454324 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:21:01.454324 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:01.454324 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:21:01.454324 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:01.455613 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:21:01.455613 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:21:01.455613 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:01.455613 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:21:01.455613 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:01.455613 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:21:01.960212 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:21:01.960839 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:21:01.961037 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:21:01.962229 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b027d63e-13a5-445b-84a0-8ae6223e05d5, routes=[], if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:21:01.962561 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:01.963411 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:21:01.963411 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:01.963411 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:21:01.963411 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:21:01.963411 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:01.963411 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:21:01.963411 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:01.963411 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:21:01.965042 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:01.965042 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:21:01.965042 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:01.965042 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:21:01.965042 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:21:01.965042 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:21:01.965042 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:21:01.965042 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:01.965042 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:01.965042 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:01.965042 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:21:01.965042 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:01.965638 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:01.965638 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:01.965638 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:21:01.965638 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:01.965638 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:21:01.965638 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:21:01.965638 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:01.965638 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:21:01.965638 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:01.965638 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:21:01.965638 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:21:01.965638 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:01.965638 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:21:01.966180 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:01.966180 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:21:01.966715 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:01.966715 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:21:01.966715 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:21:01.966715 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:21:01.966715 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:21:01.966715 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:21:01.966715 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:01.966715 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:21:01.966715 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:21:01.966715 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:01.966715 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:21:01.966715 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:21:01.966715 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:01.966715 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:01.966715 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:21:01.966715 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:01.968108 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:21:01.968108 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:21:01.968108 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:01.968108 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:01.968108 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:01.968108 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:21:01.968108 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:01.968108 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:01.968108 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:01.968108 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:21:01.968108 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:01.968108 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:21:01.968108 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:21:01.968108 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:01.968108 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:21:01.968108 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:01.968108 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:21:01.968108 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:21:01.968108 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:01.969978 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:21:01.969978 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:01.969978 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:21:01.969978 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:21:01.970287 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:21:01.970287 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:21:01.970393 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105458) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 15 10:21:01.970993 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:21:01.971283 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:01.971994 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:21:01.971994 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:01.971994 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:21:01.971994 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:21:01.971994 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:01.971994 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:21:01.971994 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:01.971994 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:21:01.972876 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:01.972876 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:21:01.972876 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:21:01.972876 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:21:01.972876 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:01.972876 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:21:01.972876 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:01.972876 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:21:01.972876 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:01.972876 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:21:01.972876 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:01.972876 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:21:01.972876 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:01.972876 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:21:01.972876 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:21:01.973945 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:01.973945 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:21:01.973945 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:01.973945 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:21:01.973945 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:21:01.973945 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:01.973945 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:21:01.973945 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:01.973945 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:21:01.973945 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:01.973945 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:21:01.973945 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:01.973945 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:21:01.973945 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:21:01.974197 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:21:01.974197 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:21:01.974197 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 15 10:21:01.974197 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:21:01.974197 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:21:01.974197 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:01.974197 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:01.974197 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:01.974197 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:21:01.974197 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:01.974197 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:01.974197 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:01.974197 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:21:01.974423 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:01.974423 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:21:01.974423 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:21:01.974423 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:01.974423 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:21:01.974423 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:01.974423 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:21:01.974423 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:21:01.974423 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:01.974423 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:21:01.974423 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:01.974423 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:21:01.974738 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:01.974738 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:21:01.974738 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:21:01.974738 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:21:01.974738 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:21:01.974738 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:21:01.974738 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:01.974738 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:21:01.974738 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:21:01.974738 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:01.974738 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:21:01.974738 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:21:01.974738 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:01.974738 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:01.974738 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:21:01.974738 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:01.975092 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:21:01.975092 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:21:01.975092 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^ May 15 10:21:01.975092 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:21:01.975092 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:21:01.975092 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:01.975092 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:01.975092 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:01.975092 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:21:01.975092 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:01.975092 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:01.975092 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:01.975092 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:21:01.975092 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:01.975092 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:21:01.975092 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:21:01.975092 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:01.975092 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:21:01.975610 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:01.975610 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:21:01.975610 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:21:01.975610 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:01.975610 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:21:01.975610 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:01.975610 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:21:01.981921 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:21:01.982275 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_ha_failover {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:21:01.982456 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:21:01.982650 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:21:01.983497 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:21:01.983969 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:01.984891 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:21:01.984891 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:01.984891 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:21:01.984891 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:21:01.984891 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:01.984891 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:21:01.984891 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:01.984891 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:21:01.986376 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:01.986376 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:21:01.986376 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:21:01.986376 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:21:01.986376 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:01.986376 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:21:01.986376 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:01.986376 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:21:01.986376 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:01.986376 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:21:01.986376 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:01.986376 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:21:01.986376 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:01.986376 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:21:01.986376 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:21:01.987114 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:01.987114 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:21:01.987114 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:01.987114 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:21:01.987114 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:21:01.987114 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:01.987114 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:21:01.987114 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:01.987114 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:21:01.988128 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_ha_failover failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:01.988128 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:21:01.988128 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:01.988128 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:21:01.988128 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:21:01.988128 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:21:01.988128 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 15 10:21:01.988128 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:21:01.988128 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:21:01.988128 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:01.988128 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:01.988777 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:01.988777 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:21:01.988777 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:01.988777 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:01.988777 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:01.988777 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:21:01.988777 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:01.988777 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:21:01.988777 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:21:01.988777 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:01.988777 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:21:01.988777 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:01.988777 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:21:01.988777 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:21:01.989319 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:01.989319 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:21:01.989319 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:01.989319 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:21:01.990072 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:01.990072 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:21:01.990072 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:21:01.990072 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:21:01.990072 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:21:01.990072 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:21:01.990072 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:01.990072 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:21:01.990072 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:21:01.990072 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:01.990072 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:21:01.990072 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:21:01.990072 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:01.990072 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:01.990072 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:21:01.990072 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:01.990072 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:21:01.990940 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 15 10:21:01.990940 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:21:01.990940 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:21:01.990940 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:01.990940 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:01.990940 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:01.990940 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:21:01.990940 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:01.990940 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:01.990940 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:01.990940 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:21:01.990940 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:01.990940 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:21:01.990940 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:21:01.990940 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:01.990940 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:21:01.990940 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:01.990940 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:21:01.990940 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:21:01.991496 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:01.991496 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:21:01.991496 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:01.991496 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:21:01.991801 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:21:01.992092 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:21:01.992207 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:21:01.992360 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:21:01.992872 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:21:01.993235 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:21:01.993478 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:01.994256 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:21:01.994256 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:01.994256 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:21:01.994256 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:21:01.994256 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:01.994256 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:21:01.994256 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:01.994256 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:21:01.995267 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:01.995267 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:21:01.995267 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:01.995267 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:21:01.995267 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:21:01.995267 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:21:01.995267 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:21:01.995267 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:01.995267 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:01.995267 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:01.995267 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:21:01.995267 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:01.995692 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:01.995692 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:01.995692 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:21:01.995692 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:01.995692 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:21:01.995692 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:21:01.995692 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:01.995692 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:21:01.995692 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:01.995692 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:21:01.995692 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:21:01.995692 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:01.995692 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:21:01.996063 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:01.996063 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:21:01.996700 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:01.996700 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:21:01.996700 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:21:01.996700 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:21:01.996700 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:21:01.996700 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:21:01.996700 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:01.996700 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:21:01.996700 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:21:01.996700 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:01.996700 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:21:01.996700 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:21:01.996700 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:01.996700 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:01.996700 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:21:01.996700 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:01.997127 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:21:01.997127 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:21:01.997127 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:01.997127 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:01.997127 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:01.997127 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:21:01.997127 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:01.997127 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:01.997127 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:01.997127 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:21:01.997127 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:01.997127 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:21:01.997127 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:21:01.997127 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:01.997127 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:21:01.997127 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:01.997127 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:21:01.997127 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:21:01.997127 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:01.997636 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:21:01.997636 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:01.997636 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:21:06.417319 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:21:06.417986 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:21:06.417986 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:21:06.418759 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:21:06.455351 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:21:06.455641 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:06.456717 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:21:06.456717 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:06.456717 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:21:06.456717 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:21:06.456717 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:06.456717 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:21:06.456717 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:06.456717 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:21:06.458018 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_fdb_aging_settings failed after 0.039 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:06.458018 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:21:06.458018 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:06.458018 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:21:06.458018 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:21:06.458018 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:21:06.458018 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:21:06.458018 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:06.458018 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:06.458018 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:06.458018 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:21:06.458018 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:06.458475 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:06.458475 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:06.458475 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:21:06.458475 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:06.458475 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:21:06.458475 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:21:06.458475 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:06.458475 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:21:06.458475 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:06.458475 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:21:06.458475 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:21:06.458475 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:06.458475 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:21:06.458995 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:06.458995 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:21:06.459363 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:06.459363 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:21:06.459363 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:21:06.459363 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:21:06.459363 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:21:06.459363 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:21:06.459363 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:06.459363 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:21:06.459363 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:21:06.459363 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:06.459363 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:21:06.459363 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:21:06.459363 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:06.459363 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:06.459363 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:21:06.459363 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:06.460040 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:21:06.460040 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:21:06.460040 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:06.460040 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:06.460040 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:06.460040 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:21:06.460040 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:06.460040 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:06.460040 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:06.460040 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:21:06.460040 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:06.460040 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:21:06.460040 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:21:06.460040 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:06.460040 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:21:06.460040 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:06.460040 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:21:06.460040 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:21:06.460040 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:06.460733 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:21:06.460733 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:06.460733 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:21:06.960928 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:21:06.961429 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:21:06.961513 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:21:06.962338 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b027d63e-13a5-445b-84a0-8ae6223e05d5, routes=[], if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:21:06.962589 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:06.963289 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:21:06.963289 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:06.963289 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:21:06.963289 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:21:06.963289 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:06.963289 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:21:06.963289 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:06.963289 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:21:06.964435 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:06.964435 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:21:06.964435 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:06.964435 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:21:06.964435 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:21:06.964435 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:21:06.964435 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:21:06.964435 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:06.964435 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:06.964435 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:06.964435 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:21:06.964435 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:06.965038 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:06.965038 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:06.965038 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:21:06.965038 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:06.965038 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:21:06.965038 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:21:06.965038 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:06.965038 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:21:06.965038 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:06.965038 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:21:06.965038 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:21:06.965038 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:06.965038 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:21:06.965730 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:06.965730 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:21:06.965730 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:06.965730 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:21:06.965730 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:21:06.965730 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:21:06.965730 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:21:06.965730 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:21:06.965730 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:06.965730 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:21:06.965730 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:21:06.965730 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:06.965730 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:21:06.965730 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:21:06.965730 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:06.966466 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:06.966466 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:21:06.966466 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:06.966466 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:21:06.966466 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:21:06.966466 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:06.966466 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:06.966466 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:06.966466 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:21:06.966466 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:06.966466 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:06.966466 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:06.966466 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:21:06.966466 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:06.966466 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:21:06.966466 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:21:06.966466 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:06.966466 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:21:06.966466 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:06.967396 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:21:06.967396 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:21:06.967396 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:06.967396 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:21:06.967396 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:06.967396 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:21:06.970113 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:21:06.970514 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:21:06.970673 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:21:06.970884 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105458) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 15 10:21:06.971637 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:21:06.971939 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:06.972779 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:21:06.972779 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:06.972779 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:21:06.972779 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:21:06.972779 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:06.972779 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:21:06.972779 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:06.972779 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:21:06.973687 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:06.973687 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:21:06.973687 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:21:06.973687 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:21:06.973687 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:06.973687 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:21:06.973687 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:06.973687 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:21:06.973687 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:06.973687 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:21:06.973687 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:06.973687 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:21:06.973687 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:06.973687 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:21:06.973687 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:21:06.974407 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:06.974407 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:21:06.974407 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:06.974407 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:21:06.974407 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:21:06.974407 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:06.974407 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:21:06.974407 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:06.974407 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:21:06.974713 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:06.974713 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:21:06.974713 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:06.974713 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:21:06.974713 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:21:06.974713 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:21:06.974713 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:21:06.974713 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 15 10:21:06.974713 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:21:06.974713 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:21:06.974713 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:06.974713 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:06.975104 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:06.975104 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:21:06.975104 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:06.975104 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:06.975104 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:06.975104 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:21:06.975104 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:06.975104 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:21:06.975104 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:21:06.975104 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:06.975104 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:21:06.975104 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:06.975104 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:21:06.975104 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:21:06.975503 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:06.975503 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:21:06.975503 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:06.975503 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:21:06.975716 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:06.975716 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:21:06.975716 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:21:06.975716 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:21:06.975716 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:21:06.975716 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:21:06.975716 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:06.975716 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:21:06.975716 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:21:06.975716 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:06.975716 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:21:06.975716 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:21:06.975716 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:06.975716 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:06.975716 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:21:06.975716 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:06.976333 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:21:06.976333 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:21:06.976333 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^ May 15 10:21:06.976333 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:21:06.976333 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:21:06.976333 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:06.976333 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:06.976333 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:06.976333 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:21:06.976333 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:06.976333 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:06.976333 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:06.976333 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:21:06.976333 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:06.976333 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:21:06.976333 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:21:06.976333 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:06.976333 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:21:06.976883 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:06.976883 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:21:06.976883 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:21:06.976883 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:06.976883 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:21:06.976883 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:06.976883 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:21:06.982473 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:21:06.982866 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_ha_failover {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:21:06.982991 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:21:06.983147 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:21:06.984089 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:21:06.984597 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:06.985837 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:21:06.985837 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:06.985837 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:21:06.985837 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:21:06.985837 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:06.985837 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:21:06.985837 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:06.985837 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:21:06.986778 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:06.986778 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:21:06.986778 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:21:06.986778 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:21:06.986778 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:06.986778 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:21:06.986778 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:06.986778 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:21:06.986778 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:06.986778 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:21:06.986778 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:06.986778 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:21:06.986778 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:06.986778 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:21:06.986778 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:21:06.987508 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:06.987508 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:21:06.987508 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:06.987508 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:21:06.987508 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:21:06.987508 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:06.987508 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:21:06.987508 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:06.987508 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:21:06.987982 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_ha_failover failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:06.987982 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:21:06.987982 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:06.987982 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:21:06.987982 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:21:06.987982 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:21:06.987982 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 15 10:21:06.987982 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:21:06.987982 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:21:06.987982 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:06.987982 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:06.988760 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:06.988760 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:21:06.988760 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:06.988760 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:06.988760 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:06.988760 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:21:06.988760 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:06.988760 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:21:06.988760 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:21:06.988760 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:06.988760 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:21:06.988760 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:06.988760 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:21:06.988760 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:21:06.989477 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:06.989477 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:21:06.989477 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:06.989477 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:21:06.989477 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:06.989477 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:21:06.989477 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:21:06.989477 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:21:06.989477 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:21:06.989477 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:21:06.989477 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:06.989477 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:21:06.989477 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:21:06.989477 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:06.990217 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:21:06.990217 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:21:06.990217 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:06.990217 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:06.990217 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:21:06.990217 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:06.990217 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:21:06.990217 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 15 10:21:06.990217 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:21:06.990217 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:21:06.990217 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:06.990217 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:06.990217 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:06.990217 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:21:06.990217 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:06.990217 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:06.990217 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:06.990217 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:21:06.990217 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:06.991008 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:21:06.991008 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:21:06.991008 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:06.991008 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:21:06.991008 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:06.991008 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:21:06.991008 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:21:06.991008 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:06.991008 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:21:06.991008 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:06.991008 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:21:06.992195 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:21:06.993138 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:21:06.993138 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:21:06.993138 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:21:06.993448 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:21:06.994170 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:21:06.994237 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:06.995342 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:21:06.995342 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:06.995342 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:21:06.995342 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:21:06.995342 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:06.995342 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:21:06.995342 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:06.995342 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:21:06.996463 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:06.996463 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:21:06.996463 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:06.996463 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:21:06.996463 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:21:06.996463 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:21:06.996463 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:21:06.996463 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:06.996463 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:06.996463 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:06.996463 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:21:06.996463 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:06.996984 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:06.996984 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:06.996984 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:21:06.996984 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:06.996984 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:21:06.996984 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:21:06.996984 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:06.996984 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:21:06.996984 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:06.996984 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:21:06.996984 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:21:06.996984 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:06.996984 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:21:06.997430 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:06.997430 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:21:06.997548 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:06.997548 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:21:06.997548 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:21:06.997548 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:21:06.997548 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:21:06.997548 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:21:06.997548 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:06.997548 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:21:06.997548 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:21:06.997548 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:06.997548 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:21:06.997548 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:21:06.997548 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:06.997548 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:06.997548 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:21:06.997548 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:06.998063 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:21:06.998063 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:21:06.998063 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:06.998063 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:06.998063 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:06.998063 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:21:06.998063 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:06.998063 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:06.998063 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:06.998063 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:21:06.998063 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:06.998063 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:21:06.998063 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:21:06.998063 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:06.998063 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:21:06.998063 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:06.998063 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:21:06.998063 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:21:06.998063 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:06.998909 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:21:06.998909 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:06.998909 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:21:11.417622 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:21:11.418481 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:21:11.418481 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:21:11.419293 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:21:11.457088 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:21:11.457572 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:11.458536 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:21:11.458536 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:11.458536 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:21:11.458536 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:21:11.458536 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:11.458536 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:21:11.458536 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:11.458536 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:21:11.459354 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_fdb_aging_settings failed after 0.041 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:11.459354 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:21:11.459354 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:11.459354 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:21:11.459354 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:21:11.459354 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:21:11.459354 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:21:11.459354 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:11.459354 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:11.459354 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:11.459354 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:21:11.459354 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:11.459769 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:11.459769 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:11.459769 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:21:11.459769 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:11.459769 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:21:11.459769 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:21:11.459769 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:11.459769 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:21:11.459769 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:11.459769 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:21:11.459769 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:21:11.459769 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:11.459769 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:21:11.460221 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:11.460221 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:21:11.460342 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:11.460342 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:21:11.460342 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:21:11.460342 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:21:11.460342 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:21:11.460342 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:21:11.460342 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:11.460342 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:21:11.460342 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:21:11.460342 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:11.460342 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:21:11.460342 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:21:11.460342 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:11.460342 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:11.460342 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:21:11.460342 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:11.461047 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:21:11.461047 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:21:11.461047 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:11.461047 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:11.461047 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:11.461047 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:21:11.461047 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:11.461047 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:11.461047 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:11.461047 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:21:11.461047 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:11.461047 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:21:11.461047 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:21:11.461047 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:11.461047 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:21:11.461047 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:11.461047 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:21:11.461047 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:21:11.461047 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:11.461715 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:21:11.461715 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:11.461715 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:21:11.961711 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:21:11.962546 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:21:11.962885 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:21:11.964078 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b027d63e-13a5-445b-84a0-8ae6223e05d5, routes=[], if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:21:11.964396 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:11.966045 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:21:11.966045 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:11.966045 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:21:11.966045 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:21:11.966045 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:11.966045 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:21:11.966045 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:11.966045 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:21:11.966456 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:11.966456 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:21:11.966456 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:11.966456 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:21:11.966456 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:21:11.966456 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:21:11.966456 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:21:11.966456 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:11.966456 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:11.966456 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:11.966456 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:21:11.966456 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:11.967266 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:11.967266 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:11.967266 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:21:11.967266 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:11.967266 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:21:11.967266 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:21:11.967266 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:11.967266 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:21:11.967266 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:11.967266 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:21:11.967266 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:21:11.967266 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:11.967266 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:21:11.967901 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:11.967901 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:21:11.967901 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:11.967901 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:21:11.967901 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:21:11.967901 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:21:11.967901 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:21:11.967901 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:21:11.967901 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:11.967901 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:21:11.967901 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:21:11.967901 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:11.967901 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:21:11.967901 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:21:11.967901 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:11.969712 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:11.969712 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:21:11.969712 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:11.969712 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:21:11.969712 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:21:11.969712 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:11.969712 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:11.969712 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:11.969712 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:21:11.969712 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:11.969712 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:11.969712 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:11.969712 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:21:11.969712 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:11.969712 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:21:11.969712 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:21:11.969712 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:11.969712 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:21:11.969712 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:11.970256 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:21:11.970256 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:21:11.970256 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:11.970256 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:21:11.970256 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:11.970256 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:21:11.970555 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:21:11.970857 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:21:11.970941 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:21:11.971146 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105458) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 15 10:21:11.971760 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:21:11.972084 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:11.972897 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:21:11.972897 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:11.972897 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:21:11.972897 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:21:11.972897 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:11.972897 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:21:11.972897 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:11.972897 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:21:11.974045 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:11.974045 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:21:11.974045 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:21:11.974045 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:21:11.974045 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:11.974045 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:21:11.974045 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:11.974045 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:21:11.974045 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:11.974045 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:21:11.974045 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:11.974045 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:21:11.974045 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:11.974045 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:21:11.974045 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:21:11.974479 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:11.974479 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:21:11.974479 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:11.974479 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:21:11.974479 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:21:11.974479 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:11.974479 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:21:11.974479 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:11.974479 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:21:11.975482 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:11.975482 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:21:11.975482 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:11.975482 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:21:11.975482 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:21:11.975482 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:21:11.975482 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:21:11.975482 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 15 10:21:11.975482 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:21:11.975482 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:21:11.975482 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:11.975482 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:11.975860 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:11.975860 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:21:11.975860 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:11.975860 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:11.975860 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:11.975860 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:21:11.975860 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:11.975860 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:21:11.975860 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:21:11.975860 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:11.975860 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:21:11.975860 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:11.975860 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:21:11.975860 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:21:11.976233 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:11.976233 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:21:11.976233 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:11.976233 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:21:11.977107 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:11.977107 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:21:11.977107 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:21:11.977107 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:21:11.977107 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:21:11.977107 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:21:11.977107 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:11.977107 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:21:11.977107 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:21:11.977107 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:11.977107 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:21:11.977107 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:21:11.977107 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:11.977107 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:11.977107 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:21:11.977107 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:11.977556 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:21:11.977556 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:21:11.977556 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^ May 15 10:21:11.977556 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:21:11.977556 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:21:11.977556 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:11.977556 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:11.977556 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:11.977556 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:21:11.977556 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:11.977556 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:11.977556 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:11.977556 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:21:11.977556 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:11.977556 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:21:11.977556 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:21:11.977556 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:11.977556 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:21:11.978029 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:11.978029 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:21:11.978029 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:21:11.978029 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:11.978029 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:21:11.978029 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:11.978029 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:21:11.983093 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:21:11.983505 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_ha_failover {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:21:11.983625 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:21:11.983779 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:21:11.984413 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:21:11.984951 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:11.985961 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:21:11.985961 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:11.985961 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:21:11.985961 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:21:11.985961 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:11.985961 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:21:11.985961 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:11.985961 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:21:11.986860 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:11.986860 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:21:11.986860 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:21:11.986860 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:21:11.986860 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:11.986860 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:21:11.986860 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:11.986860 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:21:11.986860 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:11.986860 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:21:11.986860 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:11.986860 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:21:11.986860 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:11.986860 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:21:11.986860 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:21:11.987643 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:11.987643 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:21:11.987643 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:11.987643 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:21:11.987643 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:21:11.987643 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:11.987643 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:21:11.987643 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:11.987643 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:21:11.988066 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:11.988066 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:21:11.988066 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:11.988066 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:21:11.988066 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:21:11.988066 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:21:11.988066 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 15 10:21:11.988066 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:21:11.988066 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:21:11.988066 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:11.988066 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:11.988576 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:11.988576 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:21:11.988576 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:11.988576 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:11.988576 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:11.988576 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:21:11.988576 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:11.988576 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:21:11.988576 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:21:11.988576 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:11.988576 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:21:11.988576 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:11.988576 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:21:11.988576 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:21:11.989182 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:11.989182 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:21:11.989182 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:11.989182 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:21:11.989182 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:11.989182 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:21:11.989182 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:21:11.989182 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:21:11.989182 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:21:11.989182 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:21:11.989182 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:11.989182 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:21:11.989182 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:21:11.989182 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:11.989978 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:21:11.989978 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:21:11.989978 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:11.989978 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:11.989978 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:21:11.989978 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:11.989978 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:21:11.989978 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 15 10:21:11.989978 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:21:11.989978 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:21:11.989978 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:11.989978 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:11.989978 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:11.989978 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:21:11.989978 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:11.989978 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:11.989978 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:11.989978 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:21:11.989978 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:11.990813 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:21:11.990813 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:21:11.990813 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:11.990813 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:21:11.990813 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:11.990813 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:21:11.990813 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:21:11.990813 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:11.990813 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:21:11.990813 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:11.990813 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:21:11.992569 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:21:11.992862 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:21:11.992984 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:21:11.993155 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:21:11.993926 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:21:11.994464 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:21:11.994733 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:11.995658 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:21:11.995658 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:11.995658 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:21:11.995658 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:21:11.995658 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:11.995658 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:21:11.995658 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:11.995658 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:21:11.996559 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:11.996559 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:21:11.996559 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:11.996559 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:21:11.996559 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:21:11.996559 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:21:11.996559 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:21:11.996559 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:11.996559 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:11.996559 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:11.996559 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:21:11.996559 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:11.996958 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:11.996958 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:11.996958 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:21:11.996958 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:11.996958 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:21:11.996958 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:21:11.996958 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:11.996958 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:21:11.996958 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:11.996958 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:21:11.996958 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:21:11.996958 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:11.996958 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:21:11.997310 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:11.997310 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:21:11.997780 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:11.997780 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:21:11.997780 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:21:11.997780 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:21:11.997780 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:21:11.997780 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:21:11.997780 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:11.997780 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:21:11.997780 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:21:11.997780 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:11.997780 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:21:11.997780 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:21:11.997780 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:11.997780 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:11.997780 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:21:11.997780 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:11.998211 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:21:11.998211 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:21:11.998211 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:11.998211 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:11.998211 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:11.998211 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:21:11.998211 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:11.998211 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:11.998211 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:11.998211 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:21:11.998211 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:11.998211 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:21:11.998211 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:21:11.998211 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:11.998211 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:21:11.998211 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:11.998211 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:21:11.998211 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:21:11.998211 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:11.998772 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:21:11.998772 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:11.998772 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:21:17.515712 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:21:17.515712 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:21:17.515712 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:21:17.516032 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:21:17.516032 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:21:17.520846 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:21:17.520846 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:21:17.520846 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:21:17.590567 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:21:17.590775 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:17.591539 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:21:17.591539 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:17.591539 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:21:17.591539 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:21:17.591539 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:17.591539 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:21:17.591539 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:17.591539 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:21:17.592688 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_fdb_aging_settings failed after 0.074 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:17.592688 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:21:17.592688 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:17.592688 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:21:17.592688 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:21:17.592688 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:21:17.592688 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:21:17.592688 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:17.592688 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:17.592688 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:17.592688 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:21:17.592688 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:17.593044 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:17.593044 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:17.593044 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:21:17.593044 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:17.593044 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:21:17.593044 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:21:17.593044 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:17.593044 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:21:17.593044 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:17.593044 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:21:17.593044 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:21:17.593044 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:17.593044 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:21:17.593351 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:17.593351 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:21:17.597717 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:17.597717 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:21:17.597717 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:21:17.597717 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:21:17.597717 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:21:17.597717 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:21:17.597717 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:17.597717 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:21:17.597717 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:21:17.597717 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:17.597717 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:21:17.597717 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:21:17.597717 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:17.597717 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:17.597717 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:21:17.597717 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:17.598308 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:21:17.598308 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:21:17.598308 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:17.598308 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:17.598308 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:17.598308 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:21:17.598308 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:17.598308 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:17.598308 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:17.598308 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:21:17.598308 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:17.598308 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:21:17.598308 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:21:17.598308 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:17.598308 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:21:17.598308 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:17.598308 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:21:17.598308 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:21:17.598308 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:17.598803 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:21:17.598803 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:17.598803 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:21:17.598803 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:21:17.598803 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:21:17.598803 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b027d63e-13a5-445b-84a0-8ae6223e05d5, routes=[], if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:21:17.598803 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:17.600484 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:21:17.600484 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:17.600484 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:21:17.600484 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:21:17.600484 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:17.600484 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:21:17.600484 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:17.600484 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:21:17.603299 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:17.603299 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:21:17.603299 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:17.603299 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:21:17.603299 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:21:17.603299 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:21:17.603299 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:21:17.603299 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:17.603299 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:17.603299 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:17.603299 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:21:17.603299 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:17.603887 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:17.603887 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:17.603887 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:21:17.603887 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:17.603887 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:21:17.603887 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:21:17.603887 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:17.603887 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:21:17.603887 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:17.603887 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:21:17.603887 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:21:17.603887 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:17.603887 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:21:17.604344 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:17.604344 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:21:17.604344 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:17.604344 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:21:17.604344 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:21:17.604344 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:21:17.604344 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:21:17.604344 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:21:17.604344 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:17.604344 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:21:17.604344 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:21:17.604344 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:17.604344 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:21:17.604344 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:21:17.604344 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:17.605061 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:17.605061 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:21:17.605061 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:17.605061 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:21:17.605061 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:21:17.605061 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:17.605061 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:17.605061 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:17.605061 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:21:17.605061 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:17.605061 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:17.605061 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:17.605061 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:21:17.605061 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:17.605061 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:21:17.605061 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:21:17.605061 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:17.605061 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:21:17.605061 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:17.605744 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:21:17.605744 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:21:17.605744 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:17.605744 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:21:17.605744 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:17.605744 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:21:17.605744 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:21:17.605744 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:21:17.605744 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105458) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 15 10:21:17.605744 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:21:17.606190 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:17.606190 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:21:17.606190 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:17.606190 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:21:17.606190 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:21:17.606190 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:17.606190 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:21:17.606190 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:17.606190 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:21:17.608893 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:17.608893 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:21:17.608893 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:21:17.608893 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:21:17.608893 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:17.608893 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:21:17.608893 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:17.608893 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:21:17.608893 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:17.608893 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:21:17.608893 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:17.608893 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:21:17.608893 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:17.608893 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:21:17.608893 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:21:17.609577 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:17.609577 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:21:17.609577 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:17.609577 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:21:17.609577 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:21:17.609577 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:17.609577 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:21:17.609577 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:17.609577 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:21:17.609577 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task set_fip_distributed_flag failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:17.609577 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:21:17.609577 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:17.609577 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:21:17.609577 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:21:17.609953 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:21:17.609953 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:21:17.609953 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 15 10:21:17.609953 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:21:17.609953 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:21:17.609953 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:17.609953 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:17.609953 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:17.609953 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:21:17.609953 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:17.609953 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:17.609953 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:17.609953 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:21:17.610267 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:17.610267 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:21:17.610267 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:21:17.610267 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:17.610267 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:21:17.610267 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:17.610267 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:21:17.610267 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:21:17.610267 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:17.610267 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:21:17.610267 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:17.610267 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:21:17.610538 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:17.610538 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:21:17.610538 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:21:17.610538 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:21:17.610538 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:21:17.610538 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:21:17.610538 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:17.610538 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:21:17.610538 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:21:17.610538 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:17.610538 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:21:17.610538 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:21:17.610538 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:17.610538 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:17.610538 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:21:17.610538 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:17.610903 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:21:17.610903 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:21:17.610903 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^ May 15 10:21:17.610903 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:21:17.610903 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:21:17.610903 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:17.610903 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:17.610903 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:17.610903 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:21:17.610903 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:17.610903 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:17.610903 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:17.610903 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:21:17.610903 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:17.610903 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:21:17.610903 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:21:17.610903 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:17.610903 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:21:17.612910 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:17.612910 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:21:17.612910 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:21:17.612910 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:17.612910 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:21:17.612910 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:17.612910 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:21:17.612910 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_ha_failover {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:21:17.612910 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:21:17.612910 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:21:17.613181 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:21:17.613181 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:17.618917 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:21:17.618917 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:17.618917 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:21:17.618917 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:21:17.618917 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:17.618917 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:21:17.618917 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:17.618917 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:21:17.618917 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:17.618917 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:21:17.618917 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:21:17.618917 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:21:17.618917 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:17.618917 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:21:17.618917 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:17.618917 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:21:17.618917 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:17.618917 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:21:17.619563 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:17.619563 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:21:17.619563 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:17.619563 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:21:17.619563 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:21:17.619563 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:17.619563 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:21:17.619563 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:17.619563 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:21:17.619563 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:21:17.619563 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:17.619563 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:21:17.619563 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:17.619563 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:21:17.619563 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_ha_failover failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:17.620010 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:21:17.620010 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:17.620010 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:21:17.620010 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:21:17.620010 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:21:17.620010 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 15 10:21:17.620010 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:21:17.620010 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:21:17.620010 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:17.620010 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:17.620010 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:17.620010 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:21:17.620010 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:17.620362 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:17.620362 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:17.620362 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:21:17.620362 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:17.620362 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:21:17.620362 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:21:17.620362 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:17.620362 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:21:17.620362 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:17.620362 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:21:17.620362 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:21:17.620362 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:17.620362 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:21:17.620788 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:17.620788 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:21:17.620788 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:17.620788 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:21:17.620788 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:21:17.620788 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:21:17.620788 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:21:17.620788 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:21:17.620788 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:17.620788 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:21:17.620788 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:21:17.620788 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:17.620788 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:21:17.620788 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:21:17.620788 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:17.621633 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:17.621633 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:21:17.621633 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:17.621633 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:21:17.621633 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 15 10:21:17.621633 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:21:17.621633 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:21:17.621633 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:17.621633 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:17.621633 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:17.621633 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:21:17.621633 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:17.621633 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:17.621633 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:17.621633 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:21:17.621633 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:17.621633 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:21:17.621633 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:21:17.621633 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:17.622169 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:21:17.622169 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:17.622169 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:21:17.622169 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:21:17.622169 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:17.622169 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:21:17.622169 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:17.622169 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:21:17.622169 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:21:17.622169 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:21:17.622169 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:21:17.622455 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:21:17.622455 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:21:17.622455 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:17.622455 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:21:17.622455 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:17.622455 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:21:17.622455 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:21:17.622455 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:17.622455 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:21:17.622455 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:17.622455 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:21:17.623613 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:17.623613 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:21:17.623613 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:17.623613 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:21:17.623613 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:21:17.623613 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:21:17.623613 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:21:17.623613 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:17.623613 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:17.623613 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:17.623613 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:21:17.623613 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:17.623988 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:17.623988 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:17.623988 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:21:17.623988 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:17.623988 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:21:17.623988 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:21:17.623988 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:17.623988 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:21:17.623988 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:17.623988 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:21:17.623988 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:21:17.623988 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:17.623988 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:21:17.624791 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:17.624791 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:21:17.624791 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:17.624791 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:21:17.624791 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:21:17.624791 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:21:17.624791 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:21:17.624791 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:21:17.624791 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:17.624791 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:21:17.624791 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:21:17.624791 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:17.624791 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:21:17.624791 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:21:17.624791 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:17.625776 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:17.625776 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:21:17.625776 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:17.625776 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:21:17.625776 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:21:17.625776 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:17.625776 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:17.625776 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:17.625776 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:21:17.625776 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:17.625776 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:17.625776 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:17.625776 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:21:17.625776 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:17.625776 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:21:17.625776 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:21:17.625776 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:17.625776 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:21:17.625776 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:17.627047 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:21:17.627047 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:21:17.627047 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:17.627047 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:21:17.627047 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:17.627047 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:21:22.518341 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:21:22.519285 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:21:22.519461 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:21:22.520395 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:21:22.558098 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:21:22.558599 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:22.559579 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:21:22.559579 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:22.559579 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:21:22.559579 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:21:22.559579 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:22.559579 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:21:22.559579 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:22.559579 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:21:22.561254 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_fdb_aging_settings failed after 0.040 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:22.561254 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:21:22.561254 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:22.561254 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:21:22.561254 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:21:22.561254 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:21:22.561254 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:21:22.561254 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:22.561254 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:22.561254 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:22.561254 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:21:22.561254 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:22.562120 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:22.562120 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:22.562120 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:21:22.562120 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:22.562120 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:21:22.562120 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:21:22.562120 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:22.562120 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:21:22.562120 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:22.562120 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:21:22.562120 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:21:22.562120 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:22.562120 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:21:22.563120 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:22.563120 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:21:22.563120 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:22.563120 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:21:22.563120 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:21:22.563120 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:21:22.563120 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:21:22.563120 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:21:22.563120 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:22.563120 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:21:22.563120 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:21:22.563120 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:22.563120 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:21:22.563120 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:21:22.563120 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:22.564854 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:22.564854 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:21:22.564854 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:22.564854 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:21:22.564854 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:21:22.564854 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:22.564854 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:22.564854 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:22.564854 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:21:22.564854 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:22.564854 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:22.564854 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:22.564854 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:21:22.564854 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:22.564854 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:21:22.564854 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:21:22.564854 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:22.564854 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:21:22.564854 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:22.566149 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:21:22.566149 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:21:22.566149 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:22.566149 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:21:22.566149 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:22.566149 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:21:22.597227 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:21:22.598003 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:21:22.598280 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:21:22.599134 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b027d63e-13a5-445b-84a0-8ae6223e05d5, routes=[], if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:21:22.599392 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:22.600300 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:21:22.600300 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:22.600300 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:21:22.600300 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:21:22.600300 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:22.600300 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:21:22.600300 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:22.600300 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:21:22.601775 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:22.601775 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:21:22.601775 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:22.601775 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:21:22.601775 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:21:22.601775 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:21:22.601775 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:21:22.601775 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:22.601775 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:22.601775 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:22.601775 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:21:22.601775 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:22.602395 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:22.602395 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:22.602395 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:21:22.602395 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:22.602395 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:21:22.602395 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:21:22.602395 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:22.602395 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:21:22.602395 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:22.602395 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:21:22.602395 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:21:22.602395 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:22.602395 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:21:22.603194 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:22.603194 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:21:22.603194 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:22.603194 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:21:22.603194 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:21:22.603194 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:21:22.603194 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:21:22.603194 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:21:22.603194 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:22.603194 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:21:22.603194 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:21:22.603194 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:22.603194 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:21:22.603194 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:21:22.603194 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:22.605160 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:22.605160 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:21:22.605160 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:22.605160 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:21:22.605160 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:21:22.605160 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:22.605160 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:22.605160 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:22.605160 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:21:22.605160 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:22.605160 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:22.605160 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:22.605160 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:21:22.605160 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:22.605160 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:21:22.605160 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:21:22.605160 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:22.605160 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:21:22.605160 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:22.606257 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:21:22.606257 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:21:22.606257 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:22.606257 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:21:22.606257 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:22.606257 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:21:22.606257 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:21:22.606257 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:21:22.606257 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:21:22.606257 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105458) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 15 10:21:22.606719 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:21:22.606719 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:22.606719 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:21:22.606719 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:22.606719 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:21:22.606719 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:21:22.606719 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:22.606719 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:21:22.606719 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:22.606719 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:21:22.606719 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:22.606719 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:21:22.606719 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:21:22.606719 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:21:22.607304 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:22.607304 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:21:22.607304 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:22.607304 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:21:22.607304 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:22.607304 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:21:22.607304 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:22.607304 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:21:22.607304 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:22.607304 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:21:22.607304 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:21:22.607304 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:22.607304 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:21:22.607304 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:22.607304 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:21:22.607304 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:21:22.607304 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:22.607304 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:21:22.608552 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:22.608552 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:21:22.608552 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:22.608552 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:21:22.608552 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:22.608552 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:21:22.608552 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:21:22.608552 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:21:22.608552 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:21:22.608552 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 15 10:21:22.608552 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:21:22.608552 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:21:22.609871 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:22.609871 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:22.609871 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:22.609871 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:21:22.609871 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:22.609871 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:22.609871 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:22.609871 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:21:22.609871 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:22.609871 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:21:22.609871 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:21:22.609871 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:22.609871 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:21:22.609871 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:22.610287 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:21:22.610287 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:21:22.610287 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:22.610287 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:21:22.610287 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:22.610287 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:21:22.610287 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:22.610287 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:21:22.610287 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:21:22.610287 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:21:22.610287 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:21:22.610287 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:21:22.610287 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:22.610287 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:21:22.610861 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:21:22.610861 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:22.610861 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:21:22.610861 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:21:22.610861 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:22.610861 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:22.610861 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:21:22.610861 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:22.610861 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:21:22.610861 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:21:22.610861 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^ May 15 10:21:22.610861 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:21:22.610861 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:21:22.610861 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:22.610861 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:22.610861 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:22.610861 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:21:22.610861 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:22.610861 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:22.611627 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:22.611627 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:21:22.611627 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:22.611627 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:21:22.611627 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:21:22.611627 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:22.611627 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:21:22.611627 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:22.611627 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:21:22.611627 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:21:22.611627 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:22.611627 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:21:22.611627 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:22.611627 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:21:22.611627 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:21:22.612235 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_ha_failover {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:21:22.612235 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:21:22.612235 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:21:22.612235 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:21:22.612235 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:22.612235 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:21:22.612235 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:22.612235 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:21:22.612235 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:21:22.612638 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:22.612638 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:21:22.612638 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:22.612638 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:21:22.613227 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:22.613227 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:21:22.613227 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:21:22.613227 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:21:22.613227 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:22.613227 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:21:22.613227 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:22.613227 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:21:22.613227 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:22.613227 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:21:22.613227 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:22.613227 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:21:22.613227 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:22.613227 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:21:22.613227 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:21:22.613862 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:22.613862 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:21:22.613862 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:22.613862 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:21:22.613862 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:21:22.613862 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:22.613862 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:21:22.613862 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:22.613862 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:21:22.614523 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:22.614523 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:21:22.614523 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:22.614523 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:21:22.614523 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:21:22.614523 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:21:22.614523 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 15 10:21:22.614523 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:21:22.614523 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:21:22.614523 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:22.614523 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:22.615038 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:22.615038 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:21:22.615038 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:22.615038 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:22.615038 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:22.615038 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:21:22.615038 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:22.615038 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:21:22.615038 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:21:22.615038 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:22.615038 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:21:22.615038 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:22.615038 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:21:22.615038 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:21:22.615365 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:22.615365 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:21:22.615365 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:22.615365 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:21:22.615918 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:22.615918 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:21:22.615918 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:21:22.615918 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:21:22.615918 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:21:22.615918 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:21:22.615918 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:22.615918 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:21:22.615918 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:21:22.615918 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:22.615918 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:21:22.615918 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:21:22.615918 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:22.615918 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:22.615918 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:21:22.615918 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:22.615918 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:21:22.616254 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 15 10:21:22.616254 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:21:22.616254 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:21:22.616254 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:22.616254 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:22.616254 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:22.616254 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:21:22.616254 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:22.616254 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:22.616254 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:22.616254 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:21:22.616254 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:22.616254 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:21:22.616254 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:21:22.616254 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:22.616254 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:21:22.616254 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:22.616254 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:21:22.616254 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:21:22.616635 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:22.616635 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:21:22.616635 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:22.616635 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:21:22.618665 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:21:22.618937 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:21:22.619072 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:21:22.619261 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:21:22.619657 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:21:22.619920 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:21:22.620052 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:22.620613 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:21:22.620613 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:22.620613 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:21:22.620613 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:21:22.620613 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:22.620613 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:21:22.620613 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:22.620613 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:21:22.621904 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:22.621904 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:21:22.621904 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:22.621904 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:21:22.621904 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:21:22.621904 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:21:22.621904 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:21:22.621904 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:22.621904 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:22.621904 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:22.621904 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:21:22.621904 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:22.622176 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:22.622176 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:22.622176 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:21:22.622176 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:22.622176 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:21:22.622176 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:21:22.622176 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:22.622176 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:21:22.622176 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:22.622176 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:21:22.622176 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:21:22.622176 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:22.622176 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:21:22.622413 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:22.622413 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:21:22.622582 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:22.622582 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:21:22.622582 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:21:22.622582 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:21:22.622582 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:21:22.622582 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:21:22.622582 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:22.622582 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:21:22.622582 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:21:22.622582 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:22.622582 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:21:22.622582 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:21:22.622582 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:22.622582 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:22.622582 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:21:22.622582 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:22.622907 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:21:22.622907 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:21:22.622907 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:22.622907 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:22.622907 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:22.622907 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:21:22.622907 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:22.622907 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:22.622907 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:22.622907 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:21:22.622907 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:22.622907 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:21:22.622907 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:21:22.622907 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:22.622907 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:21:22.622907 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:22.622907 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:21:22.622907 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:21:22.622907 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:22.623274 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:21:22.623274 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:22.623274 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:21:27.519344 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:21:27.520464 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:21:27.520464 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:21:27.520594 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:21:27.554079 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:21:27.554251 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:27.554701 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:21:27.554701 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:27.554701 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:21:27.554701 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:21:27.554701 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:27.554701 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:21:27.554701 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:27.554701 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:21:27.555367 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_fdb_aging_settings failed after 0.035 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:27.555367 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:21:27.555367 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:27.555367 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:21:27.555367 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:21:27.555367 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:21:27.555367 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:21:27.555367 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:27.555367 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:27.555367 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:27.555367 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:21:27.555367 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:27.556053 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:27.556053 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:27.556053 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:21:27.556053 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:27.556053 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:21:27.556053 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:21:27.556053 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:27.556053 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:21:27.556053 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:27.556053 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:21:27.556053 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:21:27.556053 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:27.556053 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:21:27.556570 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:27.556570 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:21:27.556570 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:27.556570 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:21:27.556570 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:21:27.556570 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:21:27.556570 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:21:27.556570 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:21:27.556570 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:27.556570 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:21:27.556570 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:21:27.556570 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:27.556570 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:21:27.556570 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:21:27.556570 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:27.557211 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:27.557211 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:21:27.557211 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:27.557211 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:21:27.557211 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:21:27.557211 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:27.557211 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:27.557211 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:27.557211 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:21:27.557211 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:27.557211 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:27.557211 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:27.557211 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:21:27.557211 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:27.557211 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:21:27.557211 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:21:27.557211 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:27.557211 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:21:27.557211 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:27.558003 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:21:27.558003 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:21:27.558003 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:27.558003 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:21:27.558003 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:27.558003 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:21:27.598127 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:21:27.598704 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:21:27.598955 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:21:27.599913 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b027d63e-13a5-445b-84a0-8ae6223e05d5, routes=[], if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:21:27.600219 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:27.601108 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:21:27.601108 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:27.601108 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:21:27.601108 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:21:27.601108 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:27.601108 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:21:27.601108 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:27.601108 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:21:27.602207 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:27.602207 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:21:27.602207 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:27.602207 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:21:27.602207 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:21:27.602207 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:21:27.602207 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:21:27.602207 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:27.602207 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:27.602207 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:27.602207 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:21:27.602207 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:27.602646 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:27.602646 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:27.602646 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:21:27.602646 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:27.602646 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:21:27.602646 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:21:27.602646 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:27.602646 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:21:27.602646 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:27.602646 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:21:27.602646 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:21:27.602646 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:27.602646 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:21:27.603177 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:27.603177 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:21:27.603839 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:27.603839 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:21:27.603839 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:21:27.603839 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:21:27.603839 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:21:27.603839 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:21:27.603839 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:27.603839 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:21:27.603839 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:21:27.603839 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:27.603839 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:21:27.603839 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:21:27.603839 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:27.603839 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:27.603839 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:21:27.603839 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:27.604473 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:21:27.604473 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:21:27.604473 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:27.604473 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:27.604473 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:27.604473 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:21:27.604473 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:27.604473 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:27.604473 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:27.604473 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:21:27.604473 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:27.604473 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:21:27.604473 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:21:27.604473 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:27.604473 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:21:27.604473 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:27.604473 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:21:27.604473 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:21:27.604473 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:27.605086 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:21:27.605086 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:27.605086 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:21:27.605086 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:21:27.605086 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:21:27.605086 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:21:27.605086 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105458) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 15 10:21:27.605086 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:21:27.605316 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:27.605887 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:21:27.605887 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:27.605887 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:21:27.605887 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:21:27.605887 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:27.605887 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:21:27.605887 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:27.605887 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:21:27.606822 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:27.606822 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:21:27.606822 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:21:27.606822 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:21:27.606822 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:27.606822 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:21:27.606822 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:27.606822 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:21:27.606822 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:27.606822 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:21:27.606822 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:27.606822 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:21:27.606822 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:27.606822 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:21:27.606822 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:21:27.607713 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:27.607713 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:21:27.607713 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:27.607713 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:21:27.607713 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:21:27.607713 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:27.607713 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:21:27.607713 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:27.607713 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:21:27.608106 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:27.608106 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:21:27.608106 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:27.608106 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:21:27.608106 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:21:27.608106 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:21:27.608106 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:21:27.608106 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 15 10:21:27.608106 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:21:27.608106 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:21:27.608106 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:27.608106 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:27.608539 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:27.608539 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:21:27.608539 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:27.608539 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:27.608539 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:27.608539 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:21:27.608539 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:27.608539 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:21:27.608539 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:21:27.608539 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:27.608539 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:21:27.608539 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:27.608539 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:21:27.608539 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:21:27.609072 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:27.609072 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:21:27.609072 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:27.609072 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:21:27.610137 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:27.610137 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:21:27.610137 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:21:27.610137 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:21:27.610137 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:21:27.610137 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:21:27.610137 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:27.610137 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:21:27.610137 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:21:27.610137 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:27.610137 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:21:27.610137 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:21:27.610137 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:27.610137 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:27.610137 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:21:27.610137 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:27.611469 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:21:27.611469 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:21:27.611469 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^ May 15 10:21:27.611469 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:21:27.611469 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:21:27.611469 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:27.611469 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:27.611469 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:27.611469 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:21:27.611469 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:27.611469 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:27.611469 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:27.611469 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:21:27.611469 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:27.611469 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:21:27.611469 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:21:27.611469 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:27.611469 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:21:27.612493 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:27.612493 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:21:27.612493 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:21:27.612493 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:27.612493 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:21:27.612493 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:27.612493 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:21:27.612493 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:21:27.612493 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_ha_failover {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:21:27.612493 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:21:27.613411 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:21:27.613411 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:21:27.613411 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:27.613411 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:21:27.613411 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:27.613411 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:21:27.613411 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:21:27.613411 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:27.613411 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:21:27.613411 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:27.613411 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:21:27.614236 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:27.614236 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:21:27.614236 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:21:27.614236 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:21:27.614236 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:27.614236 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:21:27.614236 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:27.614236 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:21:27.614236 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:27.614236 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:21:27.614236 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:27.614236 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:21:27.614236 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:27.614236 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:21:27.614236 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:21:27.615270 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:27.615270 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:21:27.615270 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:27.615270 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:21:27.615270 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:21:27.615270 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:27.615270 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:21:27.615270 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:27.615270 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:21:27.615270 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:27.615270 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:21:27.615270 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:27.615270 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:21:27.615270 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:21:27.616056 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:21:27.616056 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 15 10:21:27.616056 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:21:27.616056 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:21:27.616056 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:27.616056 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:27.616056 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:27.616056 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:21:27.616056 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:27.616056 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:27.616056 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:27.616056 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:21:27.616056 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:27.616615 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:21:27.616615 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:21:27.616615 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:27.616615 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:21:27.616615 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:27.616615 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:21:27.616615 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:21:27.616615 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:27.616615 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:21:27.616615 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:27.616615 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:21:27.616615 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:27.617095 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:21:27.617095 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:21:27.617095 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:21:27.617095 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:21:27.617095 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:21:27.617095 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:27.617095 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:21:27.617095 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:21:27.617095 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:27.617095 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:21:27.617095 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:21:27.617095 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:27.617095 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:27.617095 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:21:27.617095 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:27.617095 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:21:27.617095 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 15 10:21:27.617095 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:21:27.617095 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:21:27.617628 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:27.617628 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:27.617628 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:27.617628 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:21:27.617628 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:27.617628 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:27.617628 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:27.617628 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:21:27.617628 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:27.617628 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:21:27.617628 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:21:27.617628 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:27.617628 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:21:27.617628 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:27.617628 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:21:27.617628 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:21:27.617628 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:27.617628 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:21:27.617628 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:27.617628 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:21:27.618939 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:21:27.619127 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:21:27.619226 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:21:27.619391 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:21:27.619793 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:21:27.620009 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:21:27.620118 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:27.620538 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:21:27.620538 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:27.620538 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:21:27.620538 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:21:27.620538 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:27.620538 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:21:27.620538 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:27.620538 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:21:27.621249 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:27.621249 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:21:27.621249 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:27.621249 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:21:27.621249 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:21:27.621249 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:21:27.621249 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:21:27.621249 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:27.621249 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:27.621249 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:27.621249 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:21:27.621249 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:27.621581 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:27.621581 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:27.621581 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:21:27.621581 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:27.621581 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:21:27.621581 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:21:27.621581 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:27.621581 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:21:27.621581 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:27.621581 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:21:27.621581 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:21:27.621581 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:27.621581 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:21:27.621939 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:27.621939 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:21:27.622241 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:27.622241 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:21:27.622241 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:21:27.622241 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:21:27.622241 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:21:27.622241 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:21:27.622241 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:27.622241 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:21:27.622241 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:21:27.622241 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:27.622241 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:21:27.622241 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:21:27.622241 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:27.622241 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:27.622241 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:21:27.622241 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:27.622538 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:21:27.622538 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:21:27.622538 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:27.622538 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:27.622538 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:27.622538 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:21:27.622538 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:27.622538 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:27.622538 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:27.622538 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:21:27.622538 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:27.622538 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:21:27.622538 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:21:27.622538 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:27.622538 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:21:27.622538 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:27.622538 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:21:27.622538 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:21:27.622538 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:27.622892 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:21:27.622892 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:27.622892 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:21:32.520119 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:21:32.521476 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:21:32.521476 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:21:32.522454 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:21:32.556714 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:21:32.557172 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:32.558184 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:21:32.558184 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:32.558184 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:21:32.558184 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:21:32.558184 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:32.558184 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:21:32.558184 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:32.558184 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:21:32.559226 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_fdb_aging_settings failed after 0.037 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:32.559226 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:21:32.559226 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:32.559226 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:21:32.559226 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:21:32.559226 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:21:32.559226 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:21:32.559226 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:32.559226 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:32.559226 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:32.559226 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:21:32.559226 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:32.559810 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:32.559810 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:32.559810 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:21:32.559810 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:32.559810 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:21:32.559810 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:21:32.559810 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:32.559810 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:21:32.559810 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:32.559810 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:21:32.559810 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:21:32.559810 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:32.559810 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:21:32.560993 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:32.560993 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:21:32.560993 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:32.560993 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:21:32.560993 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:21:32.560993 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:21:32.560993 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:21:32.560993 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:21:32.560993 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:32.560993 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:21:32.560993 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:21:32.560993 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:32.560993 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:21:32.560993 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:21:32.560993 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:32.562132 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:32.562132 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:21:32.562132 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:32.562132 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:21:32.562132 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:21:32.562132 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:32.562132 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:32.562132 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:32.562132 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:21:32.562132 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:32.562132 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:32.562132 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:32.562132 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:21:32.562132 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:32.562132 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:21:32.562132 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:21:32.562132 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:32.562132 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:21:32.562132 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:32.563140 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:21:32.563140 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:21:32.563140 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:32.563140 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:21:32.563140 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:32.563140 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:21:32.598823 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:21:32.599437 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:21:32.599619 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:21:32.600537 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b027d63e-13a5-445b-84a0-8ae6223e05d5, routes=[], if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:21:32.600745 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:32.601455 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:21:32.601455 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:32.601455 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:21:32.601455 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:21:32.601455 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:32.601455 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:21:32.601455 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:32.601455 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:21:32.603006 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:32.603006 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:21:32.603006 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:32.603006 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:21:32.603006 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:21:32.603006 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:21:32.603006 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:21:32.603006 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:32.603006 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:32.603006 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:32.603006 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:21:32.603006 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:32.603619 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:32.603619 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:32.603619 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:21:32.603619 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:32.603619 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:21:32.603619 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:21:32.603619 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:32.603619 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:21:32.603619 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:32.603619 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:21:32.603619 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:21:32.603619 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:32.603619 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:21:32.604173 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:32.604173 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:21:32.604476 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:32.604476 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:21:32.604476 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:21:32.604476 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:21:32.604476 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:21:32.604476 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:21:32.604476 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:32.604476 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:21:32.604476 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:21:32.604476 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:32.604476 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:21:32.604476 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:21:32.604476 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:32.604476 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:32.604476 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:21:32.604476 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:32.605085 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:21:32.605085 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:21:32.605085 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:32.605085 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:32.605085 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:32.605085 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:21:32.605085 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:32.605085 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:32.605085 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:32.605085 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:21:32.605085 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:32.605085 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:21:32.605085 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:21:32.605085 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:32.605085 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:21:32.605085 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:32.605085 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:21:32.605085 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:21:32.605085 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:32.605589 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:21:32.605589 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:32.605589 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:21:32.605589 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:21:32.605589 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:21:32.605589 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:21:32.605589 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105458) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 15 10:21:32.605856 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:21:32.605945 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:32.606797 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:21:32.606797 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:32.606797 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:21:32.606797 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:21:32.606797 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:32.606797 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:21:32.606797 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:32.606797 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:21:32.608122 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:32.608122 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:21:32.608122 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:21:32.608122 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:21:32.608122 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:32.608122 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:21:32.608122 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:32.608122 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:21:32.608122 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:32.608122 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:21:32.608122 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:32.608122 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:21:32.608122 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:32.608122 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:21:32.608122 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:21:32.608767 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:32.608767 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:21:32.608767 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:32.608767 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:21:32.608767 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:21:32.608767 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:32.608767 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:21:32.608767 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:32.608767 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:21:32.609033 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:32.609033 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:21:32.609033 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:32.609033 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:21:32.609033 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:21:32.609033 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:21:32.609033 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:21:32.609033 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 15 10:21:32.609033 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:21:32.609033 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:21:32.609033 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:32.609033 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:32.609383 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:32.609383 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:21:32.609383 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:32.609383 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:32.609383 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:32.609383 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:21:32.609383 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:32.609383 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:21:32.609383 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:21:32.609383 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:32.609383 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:21:32.609383 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:32.609383 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:21:32.609383 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:21:32.609787 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:32.609787 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:21:32.609787 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:32.609787 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:21:32.610499 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:32.610499 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:21:32.610499 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:21:32.610499 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:21:32.610499 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:21:32.610499 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:21:32.610499 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:32.610499 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:21:32.610499 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:21:32.610499 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:32.610499 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:21:32.610499 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:21:32.610499 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:32.610499 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:32.610499 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:21:32.610499 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:32.611188 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:21:32.611188 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:21:32.611188 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^ May 15 10:21:32.611188 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:21:32.611188 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:21:32.611188 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:32.611188 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:32.611188 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:32.611188 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:21:32.611188 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:32.611188 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:32.611188 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:32.611188 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:21:32.611188 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:32.611188 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:21:32.611188 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:21:32.611188 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:32.611188 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:21:32.611652 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:32.611652 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:21:32.611652 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:21:32.611652 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:32.611652 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:21:32.611652 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:32.611652 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:21:32.611652 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:21:32.611652 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_ha_failover {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:21:32.611652 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:21:32.611968 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:21:32.611968 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:21:32.611968 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:32.612723 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:21:32.612723 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:32.612723 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:21:32.612723 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:21:32.612723 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:32.612723 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:21:32.612723 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:32.612723 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:21:32.612942 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:32.612942 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:21:32.612942 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:21:32.612942 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:21:32.612942 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:32.612942 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:21:32.612942 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:32.612942 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:21:32.612942 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:32.612942 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:21:32.612942 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:32.612942 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:21:32.612942 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:32.612942 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:21:32.612942 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:21:32.613305 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:32.613305 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:21:32.613305 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:32.613305 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:21:32.613305 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:21:32.613305 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:32.613305 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:21:32.613305 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:32.613305 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:21:32.613802 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:32.613802 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:21:32.613802 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:32.613802 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:21:32.613802 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:21:32.613802 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:21:32.613802 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 15 10:21:32.613802 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:21:32.613802 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:21:32.613802 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:32.613802 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:32.614445 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:32.614445 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:21:32.614445 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:32.614445 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:32.614445 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:32.614445 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:21:32.614445 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:32.614445 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:21:32.614445 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:21:32.614445 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:32.614445 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:21:32.614445 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:32.614445 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:21:32.614445 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:21:32.615154 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:32.615154 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:21:32.615154 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:32.615154 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:21:32.615154 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:32.615154 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:21:32.615154 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:21:32.615154 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:21:32.615154 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:21:32.615154 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:21:32.615154 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:32.615154 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:21:32.615154 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:21:32.615154 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:32.615619 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:21:32.615619 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:21:32.615619 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:32.615619 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:32.615619 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:21:32.615619 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:32.615619 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:21:32.615619 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 15 10:21:32.615619 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:21:32.615619 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:21:32.615619 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:32.615619 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:32.615619 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:32.615619 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:21:32.615619 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:32.615619 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:32.615619 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:32.615619 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:21:32.615619 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:32.616089 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:21:32.616089 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:21:32.616089 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:32.616089 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:21:32.616089 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:32.616089 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:21:32.616089 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:21:32.616089 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:32.616089 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:21:32.616089 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:32.616089 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:21:32.619422 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:21:32.619793 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:21:32.619939 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:21:32.620184 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:21:32.620838 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:21:32.620999 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:21:32.621211 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:32.621888 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:21:32.621888 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:32.621888 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:21:32.621888 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:21:32.621888 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:32.621888 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:21:32.621888 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:32.621888 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:21:32.623358 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:32.623358 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:21:32.623358 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:32.623358 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:21:32.623358 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:21:32.623358 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:21:32.623358 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:21:32.623358 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:32.623358 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:32.623358 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:32.623358 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:21:32.623358 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:32.623890 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:32.623890 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:32.623890 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:21:32.623890 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:32.623890 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:21:32.623890 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:21:32.623890 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:32.623890 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:21:32.623890 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:32.623890 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:21:32.623890 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:21:32.623890 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:32.623890 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:21:32.624266 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:32.624266 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:21:32.624472 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:32.624472 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:21:32.624472 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:21:32.624472 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:21:32.624472 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:21:32.624472 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:21:32.624472 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:32.624472 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:21:32.624472 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:21:32.624472 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:32.624472 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:21:32.624472 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:21:32.624472 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:32.624472 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:32.624472 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:21:32.624472 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:32.625663 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:21:32.625663 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:21:32.625663 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:32.625663 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:32.625663 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:32.625663 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:21:32.625663 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:32.625663 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:32.625663 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:32.625663 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:21:32.625663 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:32.625663 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:21:32.625663 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:21:32.625663 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:32.625663 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:21:32.625663 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:32.625663 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:21:32.625663 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:21:32.625663 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:32.626938 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:21:32.626938 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:32.626938 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:21:37.521014 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:21:37.521606 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:21:37.521860 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:21:37.523263 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:21:37.555465 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:21:37.555943 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:37.557008 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:21:37.557008 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:37.557008 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:21:37.557008 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:21:37.557008 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:37.557008 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:21:37.557008 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:37.557008 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:21:37.557783 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_fdb_aging_settings failed after 0.035 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:37.557783 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:21:37.557783 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:37.557783 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:21:37.557783 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:21:37.557783 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:21:37.557783 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:21:37.557783 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:37.557783 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:37.557783 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:37.557783 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:21:37.557783 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:37.558356 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:37.558356 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:37.558356 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:21:37.558356 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:37.558356 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:21:37.558356 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:21:37.558356 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:37.558356 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:21:37.558356 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:37.558356 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:21:37.558356 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:21:37.558356 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:37.558356 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:21:37.559027 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:37.559027 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:21:37.559027 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:37.559027 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:21:37.559027 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:21:37.559027 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:21:37.559027 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:21:37.559027 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:21:37.559027 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:37.559027 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:21:37.559027 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:21:37.559027 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:37.559027 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:21:37.559027 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:21:37.559027 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:37.560160 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:37.560160 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:21:37.560160 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:37.560160 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:21:37.560160 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:21:37.560160 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:37.560160 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:37.560160 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:37.560160 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:21:37.560160 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:37.560160 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:37.560160 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:37.560160 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:21:37.560160 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:37.560160 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:21:37.560160 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:21:37.560160 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:37.560160 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:21:37.560160 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:37.561639 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:21:37.561639 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:21:37.561639 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:37.561639 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:21:37.561639 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:37.561639 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:21:37.633936 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:21:37.633936 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:21:37.634108 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:21:37.634108 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:21:37.634325 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:21:37.634459 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:21:37.635548 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b027d63e-13a5-445b-84a0-8ae6223e05d5, routes=[], if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:21:37.635769 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:37.636283 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:21:37.636283 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:37.636283 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:21:37.636283 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:21:37.636283 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:37.636283 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:21:37.636283 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:37.636283 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:21:37.637453 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:37.637453 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:21:37.637453 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:37.637453 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:21:37.637453 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:21:37.637453 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:21:37.637453 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:21:37.637453 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:37.637453 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:37.637453 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:37.637453 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:21:37.637453 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:37.637996 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:37.637996 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:37.637996 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:21:37.637996 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:37.637996 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:21:37.637996 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:21:37.637996 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:37.637996 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:21:37.637996 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:37.637996 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:21:37.637996 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:21:37.637996 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:37.637996 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:21:37.638482 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:37.638482 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:21:37.639076 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:37.639076 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:21:37.639076 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:21:37.639076 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:21:37.639076 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:21:37.639076 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:21:37.639076 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:37.639076 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:21:37.639076 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:21:37.639076 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:37.639076 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:21:37.639076 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:21:37.639076 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:37.639076 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:37.639076 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:21:37.639076 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:37.639736 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:21:37.639736 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:21:37.639736 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:37.639736 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:37.639736 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:37.639736 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:21:37.639736 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:37.639736 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:37.639736 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:37.639736 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:21:37.639736 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:37.639736 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:21:37.639736 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:21:37.639736 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:37.639736 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:21:37.639736 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:37.639736 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:21:37.639736 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:21:37.639736 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:37.640300 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:21:37.640300 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:37.640300 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:21:37.640300 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:21:37.640300 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:21:37.640300 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105458) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 15 10:21:37.640300 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:21:37.640300 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:37.641293 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:21:37.641293 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:37.641293 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:21:37.641293 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:21:37.641293 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:37.641293 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:21:37.641293 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:37.641293 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:21:37.642271 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:37.642271 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:21:37.642271 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:21:37.642271 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:21:37.642271 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:37.642271 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:21:37.642271 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:37.642271 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:21:37.642271 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:37.642271 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:21:37.642271 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:37.642271 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:21:37.642271 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:37.642271 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:21:37.642271 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:21:37.642903 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:37.642903 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:21:37.642903 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:37.642903 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:21:37.642903 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:21:37.642903 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:37.642903 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:21:37.642903 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:37.642903 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:21:37.643645 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:37.643645 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:21:37.643645 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:37.643645 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:21:37.643645 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:21:37.643645 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:21:37.643645 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:21:37.643645 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 15 10:21:37.643645 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:21:37.643645 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:21:37.643645 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:37.643645 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:37.644835 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:37.644835 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:21:37.644835 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:37.644835 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:37.644835 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:37.644835 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:21:37.644835 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:37.644835 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:21:37.644835 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:21:37.644835 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:37.644835 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:21:37.644835 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:37.644835 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:21:37.644835 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:21:37.645710 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:37.645710 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:21:37.645710 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:37.645710 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:21:37.645710 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:37.645710 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:21:37.645710 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:21:37.645710 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:21:37.645710 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:21:37.645710 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:21:37.645710 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:37.645710 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:21:37.645710 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:21:37.645710 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:37.646082 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:21:37.646082 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:21:37.646082 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:37.646082 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:37.646082 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:21:37.646082 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:37.646082 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:21:37.646082 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:21:37.646082 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^ May 15 10:21:37.646082 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:21:37.646082 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:21:37.646082 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:37.646082 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:37.646082 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:37.646082 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:21:37.646082 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:37.646082 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:37.646082 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:37.646082 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:21:37.646415 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:37.646415 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:21:37.646415 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:21:37.646415 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:37.646415 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:21:37.646415 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:37.646415 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:21:37.646415 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:21:37.646415 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:37.646415 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:21:37.646415 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:37.646415 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:21:37.646415 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_ha_failover {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:21:37.646415 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:21:37.646659 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:21:37.646659 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:21:37.646659 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:37.647321 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:21:37.647321 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:37.647321 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:21:37.647321 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:21:37.647321 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:37.647321 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:21:37.647321 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:37.647321 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:21:37.647974 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:37.647974 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:21:37.647974 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:21:37.647974 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:21:37.647974 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:37.647974 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:21:37.647974 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:37.647974 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:21:37.647974 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:37.647974 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:21:37.647974 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:37.647974 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:21:37.647974 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:37.647974 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:21:37.647974 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:21:37.648885 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:37.648885 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:21:37.648885 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:37.648885 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:21:37.648885 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:21:37.648885 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:37.648885 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:21:37.648885 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:37.648885 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:21:37.649238 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:37.649238 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:21:37.649238 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:37.649238 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:21:37.649238 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:21:37.649238 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:21:37.649238 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 15 10:21:37.649238 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:21:37.649238 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:21:37.649238 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:37.649238 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:37.649530 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:37.649530 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:21:37.649530 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:37.649530 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:37.649530 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:37.649530 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:21:37.649530 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:37.649530 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:21:37.649530 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:21:37.649530 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:37.649530 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:21:37.649530 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:37.649530 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:21:37.649530 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:21:37.649899 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:37.649899 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:21:37.649899 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:37.649899 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:21:37.650012 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:37.650012 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:21:37.650012 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:21:37.650012 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:21:37.650012 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:21:37.650012 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:21:37.650012 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:37.650012 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:21:37.650012 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:21:37.650012 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:37.650012 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:21:37.650012 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:21:37.650012 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:37.650012 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:37.650012 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:21:37.650012 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:37.650012 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:21:37.650451 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 15 10:21:37.650451 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:21:37.650451 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:21:37.650451 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:37.650451 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:37.650451 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:37.650451 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:21:37.650451 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:37.650451 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:37.650451 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:37.650451 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:21:37.650451 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:37.650451 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:21:37.650451 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:21:37.650451 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:37.650451 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:21:37.650451 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:37.650451 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:21:37.650451 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:21:37.650817 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:37.650817 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:21:37.650817 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:37.650817 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:21:37.650817 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:21:37.650817 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:21:37.650817 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:21:37.651068 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:21:37.651518 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:21:37.651846 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:37.652591 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:21:37.652591 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:37.652591 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:21:37.652591 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:21:37.652591 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:37.652591 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:21:37.652591 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:37.652591 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:21:37.653401 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:37.653401 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:21:37.653401 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:37.653401 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:21:37.653401 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:21:37.653401 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:21:37.653401 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:21:37.653401 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:37.653401 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:37.653401 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:37.653401 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:21:37.653401 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:37.653774 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:37.653774 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:37.653774 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:21:37.653774 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:37.653774 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:21:37.653774 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:21:37.653774 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:37.653774 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:21:37.653774 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:37.653774 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:21:37.653774 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:21:37.653774 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:37.653774 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:21:37.654092 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:37.654092 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:21:37.654480 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:37.654480 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:21:37.654480 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:21:37.654480 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:21:37.654480 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:21:37.654480 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:21:37.654480 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:37.654480 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:21:37.654480 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:21:37.654480 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:37.654480 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:21:37.654480 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:21:37.654480 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:37.654480 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:37.654480 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:21:37.654480 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:37.655207 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:21:37.655207 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:21:37.655207 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:37.655207 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:37.655207 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:37.655207 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:21:37.655207 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:37.655207 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:37.655207 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:37.655207 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:21:37.655207 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:37.655207 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:21:37.655207 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:21:37.655207 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:37.655207 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:21:37.655207 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:37.655207 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:21:37.655207 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:21:37.655207 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:37.655950 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:21:37.655950 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:37.655950 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:21:42.521651 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:21:42.522618 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:21:42.522618 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:21:42.522793 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:21:42.558022 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:21:42.558313 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:42.558940 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:21:42.558940 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:42.558940 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:21:42.558940 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:21:42.558940 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:42.558940 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:21:42.558940 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:42.558940 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:21:42.560086 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_fdb_aging_settings failed after 0.037 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:42.560086 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:21:42.560086 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:42.560086 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:21:42.560086 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:21:42.560086 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:21:42.560086 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:21:42.560086 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:42.560086 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:42.560086 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:42.560086 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:21:42.560086 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:42.560436 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:42.560436 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:42.560436 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:21:42.560436 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:42.560436 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:21:42.560436 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:21:42.560436 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:42.560436 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:21:42.560436 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:42.560436 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:21:42.560436 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:21:42.560436 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:42.560436 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:21:42.561077 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:42.561077 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:21:42.561239 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:42.561239 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:21:42.561239 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:21:42.561239 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:21:42.561239 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:21:42.561239 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:21:42.561239 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:42.561239 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:21:42.561239 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:21:42.561239 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:42.561239 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:21:42.561239 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:21:42.561239 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:42.561239 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:42.561239 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:21:42.561239 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:42.562102 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:21:42.562102 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:21:42.562102 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:42.562102 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:42.562102 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:42.562102 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:21:42.562102 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:42.562102 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:42.562102 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:42.562102 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:21:42.562102 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:42.562102 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:21:42.562102 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:21:42.562102 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:42.562102 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:21:42.562102 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:42.562102 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:21:42.562102 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:21:42.562102 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:42.562590 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:21:42.562590 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:42.562590 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:21:42.634281 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:21:42.634701 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:21:42.634889 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:21:42.636262 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b027d63e-13a5-445b-84a0-8ae6223e05d5, routes=[], if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:21:42.636617 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:42.637713 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:21:42.637713 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:42.637713 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:21:42.637713 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:21:42.637713 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:42.637713 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:21:42.637713 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:42.637713 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:21:42.639698 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:42.639698 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:21:42.639698 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:42.639698 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:21:42.639698 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:21:42.639698 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:21:42.639698 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:21:42.639698 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:42.639698 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:42.639698 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:42.639698 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:21:42.639698 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:42.640513 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:42.640513 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:42.640513 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:21:42.640513 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:42.640513 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:21:42.640513 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:21:42.640513 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:42.640513 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:21:42.640513 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:42.640513 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:21:42.640513 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:21:42.640513 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:42.640513 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:21:42.642492 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:42.642492 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:21:42.642492 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:42.642492 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:21:42.642492 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:21:42.642492 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:21:42.642492 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:21:42.642492 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:21:42.642492 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:42.642492 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:21:42.642492 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:21:42.642492 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:42.642492 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:21:42.642492 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:21:42.642492 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:42.644892 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:42.644892 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:21:42.644892 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:42.644892 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:21:42.644892 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:21:42.644892 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:42.644892 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:42.644892 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:42.644892 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:21:42.644892 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:42.644892 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:42.644892 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:42.644892 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:21:42.644892 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:42.644892 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:21:42.644892 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:21:42.644892 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:42.644892 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:21:42.644892 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:42.645616 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:21:42.645616 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:21:42.645616 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:42.645616 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:21:42.645616 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:42.645616 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:21:42.645616 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:21:42.645616 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:21:42.645616 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:21:42.645616 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105458) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 15 10:21:42.646490 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:21:42.646490 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:42.646490 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:21:42.646490 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:42.646490 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:21:42.646490 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:21:42.646490 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:42.646490 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:21:42.646490 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:42.646490 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:21:42.646490 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:42.646490 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:21:42.646490 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:21:42.646490 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:21:42.647245 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:42.647245 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:21:42.647245 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:42.647245 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:21:42.647245 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:42.647245 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:21:42.647245 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:42.647245 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:21:42.647245 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:42.647245 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:21:42.647245 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:21:42.647245 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:42.647245 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:21:42.647245 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:42.647245 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:21:42.647245 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:21:42.647245 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:42.647245 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:21:42.648591 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:42.648591 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:21:42.648591 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:42.648591 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:21:42.648591 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:42.648591 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:21:42.648591 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:21:42.648591 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:21:42.648591 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:21:42.648591 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 15 10:21:42.648591 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:21:42.648591 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:21:42.649792 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:42.649792 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:42.649792 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:42.649792 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:21:42.649792 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:42.649792 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:42.649792 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:42.649792 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:21:42.649792 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:42.649792 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:21:42.649792 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:21:42.649792 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:42.649792 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:21:42.649792 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:42.650724 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:21:42.650724 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:21:42.650724 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:42.650724 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:21:42.650724 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:42.650724 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:21:42.650724 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:21:42.650724 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:42.650724 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:21:42.650724 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:21:42.650724 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:21:42.650724 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:21:42.651505 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:21:42.651505 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:42.651505 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:21:42.651505 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:21:42.651505 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:42.651505 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:21:42.651505 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:21:42.651505 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:42.651505 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:42.651505 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:21:42.651505 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:42.651505 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:21:42.651505 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:21:42.651505 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^ May 15 10:21:42.651505 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:21:42.651505 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:21:42.651505 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:42.651505 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:42.651505 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:42.652112 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:21:42.652112 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:42.652112 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:42.652112 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:42.652112 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:21:42.652112 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:42.652112 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:21:42.652112 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:21:42.652112 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:42.652112 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:21:42.652112 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:42.652112 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:21:42.652112 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:21:42.652112 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:42.652112 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:21:42.652112 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:42.652112 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:21:42.652112 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_ha_failover {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:21:42.652568 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:21:42.652568 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:21:42.652568 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:21:42.652568 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:21:42.652568 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:42.652568 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:21:42.652568 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:42.652568 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:21:42.652568 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:21:42.652865 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:42.652865 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:21:42.652865 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:42.652865 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:21:42.652865 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:42.652865 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:21:42.652865 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:21:42.652865 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:21:42.652865 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:42.652865 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:21:42.652865 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:42.652865 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:21:42.652865 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:42.652865 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:21:42.652865 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:42.652865 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:21:42.653299 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:42.653299 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:21:42.653299 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:21:42.653299 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:42.653299 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:21:42.653299 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:42.653299 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:21:42.653299 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:21:42.653299 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:42.653299 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:21:42.653299 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:42.653299 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:21:42.653640 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:42.653640 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:21:42.653640 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:42.653640 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:21:42.653640 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:21:42.653640 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:21:42.653640 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 15 10:21:42.653640 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:21:42.653640 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:21:42.653640 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:42.653640 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:42.653969 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:42.653969 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:21:42.653969 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:42.653969 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:42.653969 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:42.653969 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:21:42.653969 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:42.653969 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:21:42.653969 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:21:42.653969 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:42.653969 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:21:42.653969 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:42.653969 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:21:42.653969 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:21:42.654354 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:42.654354 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:21:42.654354 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:42.654354 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:21:42.654957 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:42.654957 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:21:42.654957 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:21:42.654957 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:21:42.654957 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:21:42.654957 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:21:42.654957 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:42.654957 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:21:42.654957 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:21:42.654957 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:42.654957 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:21:42.654957 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:21:42.654957 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:42.654957 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:42.654957 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:21:42.654957 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:42.654957 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:21:42.656038 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 15 10:21:42.656038 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:21:42.656038 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:21:42.656038 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:42.656038 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:42.656038 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:42.656038 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:21:42.656038 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:42.656038 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:42.656038 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:42.656038 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:21:42.656038 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:42.656038 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:21:42.656038 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:21:42.656038 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:42.656038 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:21:42.656038 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:42.656038 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:21:42.656038 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:21:42.656656 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:42.656656 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:21:42.656656 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:42.656656 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:21:42.656656 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:21:42.656656 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:21:42.656656 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:21:42.656656 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:21:42.656934 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:21:42.656934 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:42.656934 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:21:42.656934 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:42.656934 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:21:42.656934 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:21:42.656934 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:42.656934 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:21:42.656934 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:42.656934 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:21:42.657627 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:42.657627 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:21:42.657627 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:42.657627 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:21:42.657627 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:21:42.657627 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:21:42.657627 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:21:42.657627 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:42.657627 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:42.657627 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:42.657627 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:21:42.657627 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:42.658022 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:42.658022 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:42.658022 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:21:42.658022 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:42.658022 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:21:42.658022 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:21:42.658022 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:42.658022 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:21:42.658022 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:42.658022 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:21:42.658022 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:21:42.658022 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:42.658022 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:21:42.658343 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:42.658343 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:21:42.658914 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:42.658914 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:21:42.658914 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:21:42.658914 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:21:42.658914 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:21:42.658914 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:21:42.658914 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:42.658914 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:21:42.658914 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:21:42.658914 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:42.658914 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:21:42.658914 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:21:42.658914 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:42.658914 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:42.658914 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:21:42.658914 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:42.659404 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:21:42.659404 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:21:42.659404 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:42.659404 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:42.659404 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:42.659404 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:21:42.659404 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:42.659404 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:42.659404 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:42.659404 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:21:42.659404 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:42.659404 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:21:42.659404 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:21:42.659404 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:42.659404 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:21:42.659404 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:42.659404 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:21:42.659404 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:21:42.659404 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:42.659897 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:21:42.659897 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:42.659897 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:21:47.522306 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:21:47.525029 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:21:47.525029 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:21:47.525029 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:21:47.560584 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:21:47.560869 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:47.563597 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:21:47.563597 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:47.563597 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:21:47.563597 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:21:47.563597 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:47.563597 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:21:47.563597 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:47.563597 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:21:47.563597 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_fdb_aging_settings failed after 0.038 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:47.563597 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:21:47.563597 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:47.563597 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:21:47.563597 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:21:47.563597 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:21:47.563597 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:21:47.564992 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:47.564992 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:47.564992 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:47.564992 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:21:47.564992 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:47.564992 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:47.564992 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:47.564992 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:21:47.564992 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:47.564992 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:21:47.564992 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:21:47.564992 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:47.564992 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:21:47.564992 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:47.565613 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:21:47.565613 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:21:47.565613 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:47.565613 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:21:47.565613 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:47.565613 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:21:47.565613 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:47.565613 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:21:47.565613 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:21:47.565613 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:21:47.565613 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:21:47.565613 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:21:47.565613 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:47.565613 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:21:47.566243 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:21:47.566243 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:47.566243 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:21:47.566243 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:21:47.566243 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:47.566243 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:47.566243 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:21:47.566243 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:47.566243 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:21:47.566243 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:21:47.566243 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:47.566243 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:47.566243 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:47.566243 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:21:47.566243 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:47.566243 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:47.566243 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:47.566243 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:21:47.566243 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:47.566243 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:21:47.566243 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:21:47.567509 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:47.567509 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:21:47.567509 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:47.567509 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:21:47.567509 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:21:47.567509 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:47.567509 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:21:47.567509 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:47.567509 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:21:47.635151 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:21:47.635782 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:21:47.636024 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:21:47.636912 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b027d63e-13a5-445b-84a0-8ae6223e05d5, routes=[], if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:21:47.637230 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:47.638936 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:21:47.638936 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:47.638936 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:21:47.638936 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:21:47.638936 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:47.638936 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:21:47.638936 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:47.638936 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:21:47.686215 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:47.686215 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:21:47.686215 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:47.686215 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:21:47.686215 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:21:47.686215 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:21:47.686215 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:21:47.686215 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:47.686215 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:47.686215 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:47.686215 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:21:47.686215 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:47.688785 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:47.688785 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:47.688785 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:21:47.688785 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:47.688785 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:21:47.688785 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:21:47.688785 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:47.688785 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:21:47.688785 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:47.688785 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:21:47.688785 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:21:47.688785 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:47.688785 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:21:47.690126 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:47.690126 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:21:47.690126 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:47.690126 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:21:47.690126 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:21:47.690126 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:21:47.690126 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:21:47.690126 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:21:47.690126 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:47.690126 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:21:47.690126 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:21:47.690126 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:47.690126 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:21:47.690126 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:21:47.690126 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:47.691904 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:47.691904 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:21:47.691904 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:47.691904 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:21:47.691904 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:21:47.691904 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:47.691904 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:47.691904 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:47.691904 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:21:47.691904 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:47.691904 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:47.691904 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:47.691904 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:21:47.691904 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:47.691904 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:21:47.691904 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:21:47.691904 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:47.691904 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:21:47.691904 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:47.693076 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:21:47.693076 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:21:47.693076 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:47.693076 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:21:47.693076 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:47.693076 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:21:47.693076 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:21:47.693076 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:21:47.693076 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:21:47.693076 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105458) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 15 10:21:47.693703 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:21:47.693703 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:47.693703 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:21:47.693703 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:47.693703 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:21:47.693703 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:21:47.693703 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:47.693703 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:21:47.693703 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:47.693703 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:21:47.693703 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:47.693703 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:21:47.693703 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:21:47.693703 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:21:47.694608 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:47.694608 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:21:47.694608 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:47.694608 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:21:47.694608 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:47.694608 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:21:47.694608 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:47.694608 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:21:47.694608 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:47.694608 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:21:47.694608 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:21:47.694608 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:47.694608 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:21:47.694608 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:47.694608 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:21:47.694608 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:21:47.694608 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:47.694608 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:21:47.695823 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:47.695823 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:21:47.695823 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:47.695823 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:21:47.695823 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:47.695823 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:21:47.695823 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:21:47.695823 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:21:47.695823 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:21:47.695823 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 15 10:21:47.695823 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:21:47.695823 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:21:47.696700 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:47.696700 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:47.696700 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:47.696700 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:21:47.696700 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:47.696700 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:47.696700 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:47.696700 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:21:47.696700 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:47.696700 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:21:47.696700 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:21:47.696700 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:47.696700 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:21:47.696700 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:47.697815 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:21:47.697815 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:21:47.697815 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:47.697815 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:21:47.697815 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:47.697815 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:21:47.697815 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:47.697815 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:21:47.697815 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:21:47.697815 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:21:47.697815 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:21:47.697815 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:21:47.697815 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:47.697815 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:21:47.698499 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:21:47.698499 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:47.698499 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:21:47.698499 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:21:47.698499 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:47.698499 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:47.698499 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:21:47.698499 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:47.698499 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:21:47.698499 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:21:47.698499 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^ May 15 10:21:47.698499 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:21:47.698499 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:21:47.698499 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:47.698499 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:47.698499 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:47.698499 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:21:47.698499 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:47.698499 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:47.698933 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:47.698933 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:21:47.698933 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:47.698933 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:21:47.698933 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:21:47.698933 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:47.698933 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:21:47.698933 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:47.698933 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:21:47.698933 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:21:47.698933 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:47.698933 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:21:47.698933 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:47.698933 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:21:47.698933 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:21:47.699247 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_ha_failover {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:21:47.699247 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:21:47.699247 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:21:47.699247 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:21:47.699247 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:47.699247 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:21:47.699247 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:47.699247 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:21:47.699247 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:21:47.699483 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:47.699483 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:21:47.699483 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:47.699483 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:21:47.699483 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:47.699483 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:21:47.699483 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:21:47.699483 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:21:47.699483 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:47.699483 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:21:47.699483 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:47.699483 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:21:47.699483 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:47.699483 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:21:47.699483 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:47.699483 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:21:47.699842 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:47.699842 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:21:47.699842 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:21:47.699842 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:47.699842 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:21:47.699842 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:47.699842 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:21:47.699842 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:21:47.699842 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:47.699842 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:21:47.699842 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:47.699842 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:21:47.699842 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:47.699842 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:21:47.700181 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:47.700181 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:21:47.700181 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:21:47.700181 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:21:47.700181 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 15 10:21:47.700181 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:21:47.700181 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:21:47.700181 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:47.700181 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:47.700181 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:47.700181 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:21:47.700181 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:47.700181 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:47.700501 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:47.700501 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:21:47.700501 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:47.700501 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:21:47.700501 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:21:47.700501 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:47.700501 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:21:47.700501 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:47.700501 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:21:47.700501 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:21:47.700501 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:47.700501 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:21:47.700501 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:47.700808 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:21:47.700808 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:47.700808 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:21:47.700808 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:21:47.700808 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:21:47.700808 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:21:47.700808 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:21:47.700808 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:47.700808 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:21:47.700808 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:21:47.700808 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:47.700808 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:21:47.700808 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:21:47.700808 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:47.700808 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:47.700808 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:21:47.700808 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:47.701187 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:21:47.701187 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 15 10:21:47.701187 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:21:47.701187 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:21:47.701187 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:47.701187 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:47.701187 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:47.701187 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:21:47.701187 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:47.701187 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:47.701187 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:47.701187 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:21:47.701187 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:47.701187 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:21:47.701187 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:21:47.701187 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:47.701187 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:21:47.701187 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:47.701187 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:21:47.701597 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:21:47.701597 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:47.701597 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:21:47.701597 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:47.701597 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:21:47.701597 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:21:47.701597 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:21:47.701597 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:21:47.701597 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:21:47.701880 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:21:47.701880 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:21:47.701880 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:47.701880 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:21:47.701880 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:47.701880 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:21:47.701880 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:21:47.701880 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:47.701880 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:21:47.701880 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:47.701880 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:21:47.701880 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:47.702272 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:21:47.702272 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:47.702272 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:21:47.702272 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:21:47.702272 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:21:47.702272 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:21:47.702272 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:47.702272 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:47.702272 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:47.702272 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:21:47.702272 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:47.702272 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:47.702272 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:47.702272 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:21:47.702572 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:47.702572 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:21:47.702572 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:21:47.702572 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:47.702572 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:21:47.702572 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:47.702572 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:21:47.702572 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:21:47.702572 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:47.702572 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:21:47.702572 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:47.702572 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:21:47.702822 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:47.702822 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:21:47.702822 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:21:47.702822 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:21:47.702822 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:21:47.702822 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:21:47.702822 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:47.702822 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:21:47.702822 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:21:47.702822 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:47.702822 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:21:47.702822 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:21:47.702822 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:47.702822 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:47.702822 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:21:47.702822 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:47.703135 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:21:47.703135 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:21:47.703135 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:47.703135 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:47.703135 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:47.703135 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:21:47.703135 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:47.703135 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:47.703135 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:47.703135 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:21:47.703135 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:47.703135 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:21:47.703135 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:21:47.703135 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:47.703135 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:21:47.703135 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:47.703135 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:21:47.703135 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:21:47.703135 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:47.703484 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:21:47.703484 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:47.703484 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:21:52.523264 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:21:52.523746 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:21:52.523913 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:21:52.525156 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:21:52.527228 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG dbcounter [-] [105458] Writing DB stats neutron:SELECT=240 {{(pid=105458) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} May 15 10:21:52.561543 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:21:52.561877 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:52.562441 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:21:52.562441 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:52.562441 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:21:52.562441 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:21:52.562441 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:52.562441 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:21:52.562441 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:52.562441 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:21:52.563238 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_fdb_aging_settings failed after 0.039 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:52.563238 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:21:52.563238 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:52.563238 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:21:52.563238 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:21:52.563238 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:21:52.563238 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:21:52.563238 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:52.563238 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:52.563238 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:52.563238 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:21:52.563238 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:52.563933 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:52.563933 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:52.563933 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:21:52.563933 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:52.563933 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:21:52.563933 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:21:52.563933 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:52.563933 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:21:52.563933 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:52.563933 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:21:52.563933 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:21:52.563933 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:52.563933 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:21:52.564415 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:52.564415 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:21:52.564415 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:52.564415 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:21:52.564415 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:21:52.564415 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:21:52.564415 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:21:52.564415 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:21:52.564415 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:52.564415 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:21:52.564415 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:21:52.564415 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:52.564415 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:21:52.564415 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:21:52.564415 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:52.565081 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:52.565081 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:21:52.565081 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:52.565081 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:21:52.565081 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:21:52.565081 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:52.565081 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:52.565081 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:52.565081 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:21:52.565081 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:52.565081 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:52.565081 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:52.565081 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:21:52.565081 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:52.565081 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:21:52.565081 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:21:52.565081 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:52.565081 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:21:52.565081 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:52.565795 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:21:52.565795 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:21:52.565795 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:52.565795 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:21:52.565795 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:52.565795 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:21:52.635981 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:21:52.636561 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:21:52.637052 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:21:52.638025 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b027d63e-13a5-445b-84a0-8ae6223e05d5, routes=[], if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:21:52.638526 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:52.639850 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:21:52.639850 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:52.639850 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:21:52.639850 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:21:52.639850 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:52.639850 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:21:52.639850 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:52.639850 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:21:52.641528 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:52.641528 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:21:52.641528 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:52.641528 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:21:52.641528 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:21:52.641528 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:21:52.641528 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:21:52.641528 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:52.641528 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:52.641528 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:52.641528 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:21:52.641528 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:52.642137 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:52.642137 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:52.642137 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:21:52.642137 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:52.642137 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:21:52.642137 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:21:52.642137 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:52.642137 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:21:52.642137 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:52.642137 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:21:52.642137 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:21:52.642137 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:52.642137 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:21:52.642641 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:52.642641 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:21:52.642641 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:52.642641 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:21:52.642641 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:21:52.642641 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:21:52.642641 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:21:52.642641 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:21:52.642641 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:52.642641 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:21:52.642641 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:21:52.642641 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:52.642641 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:21:52.642641 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:21:52.642641 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:52.643249 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:52.643249 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:21:52.643249 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:52.643249 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:21:52.643249 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:21:52.643249 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:52.643249 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:52.643249 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:52.643249 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:21:52.643249 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:52.643249 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:52.643249 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:52.643249 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:21:52.643249 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:52.643249 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:21:52.643249 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:21:52.643249 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:52.643249 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:21:52.643249 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:52.644017 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:21:52.644017 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:21:52.644017 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:52.644017 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:21:52.644017 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:52.644017 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:21:52.644017 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:21:52.644017 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:21:52.644017 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:21:52.644017 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105458) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 15 10:21:52.644456 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:21:52.644456 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:52.645570 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:21:52.645570 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:52.645570 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:21:52.645570 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:21:52.645570 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:52.645570 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:21:52.645570 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:52.645570 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:21:52.646106 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:52.646106 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:21:52.646106 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:21:52.646106 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:21:52.646106 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:52.646106 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:21:52.646106 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:52.646106 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:21:52.646106 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:52.646106 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:21:52.646106 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:52.646106 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:21:52.646106 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:52.646106 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:21:52.646106 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:21:52.647174 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:52.647174 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:21:52.647174 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:52.647174 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:21:52.647174 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:21:52.647174 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:52.647174 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:21:52.647174 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:52.647174 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:21:52.647174 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:52.647174 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:21:52.647174 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:52.647174 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:21:52.647174 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:21:52.648205 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:21:52.648205 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:21:52.648205 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 15 10:21:52.648205 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:21:52.648205 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:21:52.648205 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:52.648205 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:52.648205 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:52.648205 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:21:52.648205 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:52.648205 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:52.648205 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:52.648205 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:21:52.649096 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:52.649096 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:21:52.649096 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:21:52.649096 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:52.649096 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:21:52.649096 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:52.649096 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:21:52.649096 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:21:52.649096 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:52.649096 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:21:52.649096 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:52.649096 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:21:52.650003 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:52.650003 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:21:52.650003 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:21:52.650003 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:21:52.650003 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:21:52.650003 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:21:52.650003 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:52.650003 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:21:52.650003 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:21:52.650003 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:52.650003 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:21:52.650003 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:21:52.650003 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:52.650003 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:52.650003 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:21:52.650003 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:52.650882 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:21:52.650882 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:21:52.650882 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^ May 15 10:21:52.650882 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:21:52.650882 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:21:52.650882 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:52.650882 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:52.650882 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:52.650882 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:21:52.650882 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:52.650882 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:52.650882 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:52.650882 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:21:52.650882 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:52.650882 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:21:52.650882 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:21:52.650882 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:52.650882 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:21:52.651523 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:52.651523 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:21:52.651523 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:21:52.651523 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:52.651523 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:21:52.651523 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:52.651523 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:21:52.651523 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:21:52.651523 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_ha_failover {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:21:52.651523 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:21:52.651907 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:21:52.651907 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:21:52.651907 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:52.651907 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:21:52.651907 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:52.651907 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:21:52.651907 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:21:52.651907 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:52.651907 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:21:52.651907 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:52.651907 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:21:52.652809 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:52.652809 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:21:52.652809 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:21:52.652809 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:21:52.652809 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:52.652809 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:21:52.652809 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:52.652809 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:21:52.652809 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:52.652809 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:21:52.652809 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:52.652809 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:21:52.652809 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:52.652809 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:21:52.652809 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:21:52.653226 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:52.653226 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:21:52.653226 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:52.653226 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:21:52.653226 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:21:52.653226 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:52.653226 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:21:52.653226 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:52.653226 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:21:52.653766 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:52.653766 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:21:52.653766 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:52.653766 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:21:52.653766 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:21:52.653766 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:21:52.653766 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 15 10:21:52.653766 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:21:52.653766 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:21:52.653766 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:52.653766 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:52.654007 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:52.654007 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:21:52.654007 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:52.654007 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:52.654007 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:52.654007 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:21:52.654007 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:52.654007 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:21:52.654007 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:21:52.654007 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:52.654007 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:21:52.654007 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:52.654007 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:21:52.654007 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:21:52.654278 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:52.654278 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:21:52.654278 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:52.654278 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:21:52.655121 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:52.655121 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:21:52.655121 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:21:52.655121 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:21:52.655121 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:21:52.655121 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:21:52.655121 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:52.655121 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:21:52.655121 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:21:52.655121 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:52.655121 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:21:52.655121 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:21:52.655121 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:52.655121 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:52.655121 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:21:52.655121 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:52.655121 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:21:52.655443 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 15 10:21:52.655443 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:21:52.655443 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:21:52.655443 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:52.655443 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:52.655443 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:52.655443 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:21:52.655443 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:52.655443 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:52.655443 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:52.655443 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:21:52.655443 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:52.655443 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:21:52.655443 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:21:52.655443 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:52.655443 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:21:52.655443 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:52.655443 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:21:52.655443 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:21:52.655850 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:52.655850 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:21:52.655850 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:52.655850 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:21:52.655850 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:21:52.655850 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:21:52.655850 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:21:52.655997 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:21:52.656312 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:21:52.656623 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:21:52.656769 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:52.657271 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:21:52.657271 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:52.657271 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:21:52.657271 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:21:52.657271 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:52.657271 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:21:52.657271 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:52.657271 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:21:52.658126 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:52.658126 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:21:52.658126 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:52.658126 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:21:52.658126 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:21:52.658126 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:21:52.658126 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:21:52.658126 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:52.658126 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:52.658126 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:52.658126 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:21:52.658126 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:52.658385 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:52.658385 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:52.658385 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:21:52.658385 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:52.658385 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:21:52.658385 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:21:52.658385 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:52.658385 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:21:52.658385 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:52.658385 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:21:52.658385 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:21:52.658385 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:52.658385 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:21:52.658616 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:52.658616 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:21:52.659216 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:52.659216 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:21:52.659216 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:21:52.659216 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:21:52.659216 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:21:52.659216 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:21:52.659216 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:52.659216 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:21:52.659216 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:21:52.659216 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:52.659216 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:21:52.659216 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:21:52.659216 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:52.659216 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:52.659216 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:21:52.659216 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:52.659525 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:21:52.659525 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:21:52.659525 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:52.659525 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:52.659525 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:52.659525 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:21:52.659525 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:52.659525 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:52.659525 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:52.659525 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:21:52.659525 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:52.659525 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:21:52.659525 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:21:52.659525 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:52.659525 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:21:52.659525 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:52.659525 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:21:52.659525 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:21:52.659525 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:52.659912 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:21:52.659912 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:52.659912 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:21:57.524212 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:21:57.525652 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:21:57.525652 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:21:57.526774 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:21:57.568575 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:21:57.569175 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:57.570310 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:21:57.570310 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:57.570310 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:21:57.570310 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:21:57.570310 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:57.570310 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:21:57.570310 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:57.570310 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:21:57.571417 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_fdb_aging_settings failed after 0.045 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:57.571417 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:21:57.571417 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:57.571417 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:21:57.571417 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:21:57.571417 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:21:57.571417 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:21:57.571417 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:57.571417 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:57.571417 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:57.571417 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:21:57.571417 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:57.572937 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:57.572937 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:57.572937 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:21:57.572937 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:57.572937 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:21:57.572937 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:21:57.572937 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:57.572937 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:21:57.572937 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:57.572937 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:21:57.572937 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:21:57.572937 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:57.572937 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:21:57.573705 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:57.573705 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:21:57.573705 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:57.573705 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:21:57.573705 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:21:57.573705 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:21:57.573705 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:21:57.573705 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:21:57.573705 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:57.573705 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:21:57.573705 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:21:57.573705 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:57.573705 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:21:57.573705 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:21:57.573705 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:57.574524 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:57.574524 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:21:57.574524 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:57.574524 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:21:57.574524 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:21:57.574524 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:57.574524 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:57.574524 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:57.574524 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:21:57.574524 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:57.574524 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:57.574524 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:57.574524 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:21:57.574524 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:57.574524 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:21:57.574524 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:21:57.574524 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:57.574524 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:21:57.574524 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:57.577026 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:21:57.577026 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:21:57.577026 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:57.577026 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:21:57.577026 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:57.577026 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:21:57.636889 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:21:57.637635 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:21:57.637905 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:21:57.638997 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b027d63e-13a5-445b-84a0-8ae6223e05d5, routes=[], if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:21:57.639320 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:57.640266 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:21:57.640266 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:57.640266 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:21:57.640266 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:21:57.640266 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:57.640266 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:21:57.640266 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:57.640266 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:21:57.641955 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:57.641955 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:21:57.641955 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:57.641955 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:21:57.641955 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:21:57.641955 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:21:57.641955 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:21:57.641955 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:57.641955 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:57.641955 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:57.641955 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:21:57.641955 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:57.643182 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:57.643182 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:57.643182 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:21:57.643182 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:57.643182 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:21:57.643182 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:21:57.643182 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:57.643182 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:21:57.643182 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:57.643182 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:21:57.643182 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:21:57.643182 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:57.643182 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:21:57.644144 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:57.644144 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:21:57.644144 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:57.644144 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:21:57.644144 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:21:57.644144 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:21:57.644144 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:21:57.644144 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:21:57.644144 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:57.644144 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:21:57.644144 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:21:57.644144 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:57.644144 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:21:57.644144 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:21:57.644144 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:57.644907 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:57.644907 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:21:57.644907 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:57.644907 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:21:57.644907 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:21:57.644907 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:57.644907 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:57.644907 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:57.644907 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:21:57.644907 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:57.644907 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:57.644907 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:57.644907 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:21:57.644907 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:57.644907 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:21:57.644907 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:21:57.644907 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:57.644907 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:21:57.644907 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:57.645487 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:21:57.645487 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:21:57.645487 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:57.645487 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:21:57.645487 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:57.645487 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:21:57.645487 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:21:57.645487 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:21:57.645487 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:21:57.645487 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105458) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 15 10:21:57.646223 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:21:57.646488 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:57.647396 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:21:57.647396 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:57.647396 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:21:57.647396 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:21:57.647396 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:57.647396 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:21:57.647396 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:57.647396 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:21:57.648853 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:57.648853 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:21:57.648853 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:21:57.648853 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:21:57.648853 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:57.648853 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:21:57.648853 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:57.648853 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:21:57.648853 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:57.648853 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:21:57.648853 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:57.648853 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:21:57.648853 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:57.648853 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:21:57.648853 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:21:57.649506 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:57.649506 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:21:57.649506 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:57.649506 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:21:57.649506 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:21:57.649506 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:57.649506 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:21:57.649506 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:57.649506 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:21:57.649811 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:57.649811 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:21:57.649811 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:57.649811 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:21:57.649811 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:21:57.649811 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:21:57.649811 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:21:57.649811 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 15 10:21:57.649811 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:21:57.649811 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:21:57.649811 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:57.649811 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:57.650507 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:57.650507 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:21:57.650507 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:57.650507 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:57.650507 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:57.650507 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:21:57.650507 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:57.650507 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:21:57.650507 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:21:57.650507 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:57.650507 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:21:57.650507 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:57.650507 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:21:57.650507 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:21:57.651170 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:57.651170 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:21:57.651170 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:57.651170 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:21:57.651392 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:57.651392 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:21:57.651392 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:21:57.651392 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:21:57.651392 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:21:57.651392 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:21:57.651392 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:57.651392 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:21:57.651392 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:21:57.651392 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:57.651392 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:21:57.651392 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:21:57.651392 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:57.651392 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:57.651392 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:21:57.651392 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:57.652431 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:21:57.652431 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:21:57.652431 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^ May 15 10:21:57.652431 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:21:57.652431 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:21:57.652431 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:57.652431 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:57.652431 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:57.652431 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:21:57.652431 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:57.652431 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:57.652431 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:57.652431 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:21:57.652431 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:57.652431 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:21:57.652431 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:21:57.652431 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:57.652431 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:21:57.653793 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:57.653793 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:21:57.653793 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:21:57.653793 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:57.653793 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:21:57.653793 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:57.653793 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:21:57.653793 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:21:57.653793 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_ha_failover {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:21:57.653793 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:21:57.654578 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:21:57.654578 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:21:57.654578 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:57.654578 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:21:57.654578 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:57.654578 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:21:57.654578 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:21:57.654578 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:57.654578 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:21:57.654578 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:57.654578 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:21:57.655188 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:57.655188 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:21:57.655188 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:21:57.655188 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:21:57.655188 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:57.655188 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:21:57.655188 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:57.655188 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:21:57.655188 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:57.655188 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:21:57.655188 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:57.655188 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:21:57.655188 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:57.655188 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:21:57.655188 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:21:57.656091 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:57.656091 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:21:57.656091 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:57.656091 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:21:57.656091 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:21:57.656091 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:57.656091 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:21:57.656091 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:57.656091 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:21:57.656352 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:57.656352 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:21:57.656352 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:57.656352 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:21:57.656352 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:21:57.656352 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:21:57.656352 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 15 10:21:57.656352 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:21:57.656352 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:21:57.656352 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:57.656352 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:57.656868 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:57.656868 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:21:57.656868 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:57.656868 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:57.656868 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:57.656868 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:21:57.656868 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:57.656868 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:21:57.656868 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:21:57.656868 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:57.656868 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:21:57.656868 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:57.656868 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:21:57.656868 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:21:57.657792 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:57.657792 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:21:57.657792 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:57.657792 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:21:57.657792 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:21:57.657792 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:57.657792 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:21:57.657792 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:21:57.657792 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:21:57.657792 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:21:57.657792 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:21:57.657792 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:57.658545 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:21:57.658545 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:21:57.658545 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:57.658545 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:21:57.658545 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:21:57.658545 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:57.658545 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:57.658545 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:21:57.658545 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:57.658545 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:21:57.658545 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 15 10:21:57.658545 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:21:57.658545 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:21:57.658545 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:57.658545 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:57.658545 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:57.658545 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:21:57.658545 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:57.658545 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:57.659534 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:57.659534 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:21:57.659534 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:57.659534 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:21:57.659534 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:21:57.659534 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:57.659534 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:21:57.659534 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:57.659534 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:21:57.659534 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:21:57.659534 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:57.659534 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:21:57.659534 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:57.659534 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:21:57.659534 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:21:57.660026 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:21:57.660026 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:21:57.660026 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:21:57.660026 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:21:57.660026 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:57.660026 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:21:57.660026 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:57.660026 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:21:57.660026 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:21:57.660343 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:57.660343 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:21:57.660343 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:57.660343 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:21:57.660583 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:57.660583 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:21:57.660583 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:57.660583 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:21:57.660583 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:21:57.660583 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:21:57.660583 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:21:57.660583 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:57.660583 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:57.660583 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:57.660583 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:21:57.660583 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:57.661326 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:21:57.661326 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:57.661326 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:21:57.661326 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:57.661326 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:21:57.661326 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:21:57.661326 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:57.661326 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:21:57.661326 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:57.661326 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:21:57.661326 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:21:57.661326 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:57.661326 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:21:57.662069 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:57.662069 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:21:57.662069 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:57.662069 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:21:57.662069 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:21:57.662069 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:21:57.662069 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:21:57.662069 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:21:57.662069 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:57.662069 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:21:57.662069 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:21:57.662069 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:57.662069 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:21:57.662069 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:21:57.662069 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:21:57.662979 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:21:57.662979 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:21:57.662979 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:21:57.662979 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:21:57.662979 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:21:57.662979 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:57.662979 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:57.662979 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:21:57.662979 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:21:57.662979 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:21:57.662979 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:21:57.662979 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:21:57.662979 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:21:57.662979 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:21:57.662979 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:21:57.662979 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:21:57.662979 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:21:57.662979 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:21:57.662979 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:21:57.664100 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:21:57.664100 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:21:57.664100 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:21:57.664100 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:21:57.664100 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:21:57.664100 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:22:02.524892 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:22:02.525747 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:22:02.526237 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:22:02.527359 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:22:02.560083 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:22:02.560571 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:02.561455 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:22:02.561455 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:02.561455 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:22:02.561455 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:22:02.561455 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:02.561455 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:22:02.561455 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:02.561455 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:22:02.562475 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_fdb_aging_settings failed after 0.035 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:02.562475 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:22:02.562475 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:02.562475 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:22:02.562475 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:22:02.562475 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:22:02.562475 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:22:02.562475 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:02.562475 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:02.562475 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:02.562475 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:22:02.562475 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:02.562984 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:02.562984 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:02.562984 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:22:02.562984 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:02.562984 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:22:02.562984 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:22:02.562984 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:02.562984 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:22:02.562984 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:02.562984 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:22:02.562984 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:22:02.562984 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:02.562984 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:22:02.563600 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:02.563600 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:22:02.563600 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:02.563600 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:22:02.563600 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:22:02.563600 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:22:02.563600 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:22:02.563600 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:22:02.563600 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:02.563600 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:22:02.563600 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:22:02.563600 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:02.563600 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:22:02.563600 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:22:02.563600 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:02.564417 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:02.564417 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:22:02.564417 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:02.564417 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:22:02.564417 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:22:02.564417 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:02.564417 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:02.564417 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:02.564417 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:22:02.564417 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:02.564417 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:02.564417 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:02.564417 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:22:02.564417 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:02.564417 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:22:02.564417 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:22:02.564417 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:02.564417 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:22:02.564417 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:02.565399 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:22:02.565399 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:22:02.565399 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:02.565399 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:22:02.565399 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:02.565399 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:22:02.637716 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:22:02.638319 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:22:02.638517 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:22:02.639458 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b027d63e-13a5-445b-84a0-8ae6223e05d5, routes=[], if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:22:02.639863 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:02.640876 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:22:02.640876 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:02.640876 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:22:02.640876 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:22:02.640876 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:02.640876 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:22:02.640876 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:02.640876 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:22:02.642341 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:02.642341 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:22:02.642341 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:02.642341 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:22:02.642341 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:22:02.642341 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:22:02.642341 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:22:02.642341 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:02.642341 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:02.642341 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:02.642341 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:22:02.642341 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:02.642698 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:02.642698 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:02.642698 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:22:02.642698 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:02.642698 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:22:02.642698 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:22:02.642698 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:02.642698 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:22:02.642698 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:02.642698 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:22:02.642698 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:22:02.642698 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:02.642698 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:22:02.643067 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:02.643067 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:22:02.644175 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:02.644175 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:22:02.644175 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:22:02.644175 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:22:02.644175 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:22:02.644175 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:22:02.644175 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:02.644175 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:22:02.644175 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:22:02.644175 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:02.644175 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:22:02.644175 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:22:02.644175 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:02.644175 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:02.644175 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:22:02.644175 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:02.644622 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:22:02.644622 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:22:02.644622 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:02.644622 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:02.644622 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:02.644622 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:22:02.644622 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:02.644622 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:02.644622 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:02.644622 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:22:02.644622 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:02.644622 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:22:02.644622 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:22:02.644622 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:02.644622 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:22:02.644622 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:02.644622 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:22:02.644622 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:22:02.644622 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:02.645178 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:22:02.645178 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:02.645178 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:22:02.645178 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:22:02.645300 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:22:02.645430 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:22:02.645554 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105458) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 15 10:22:02.646064 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:22:02.646367 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:02.647238 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:22:02.647238 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:02.647238 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:22:02.647238 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:22:02.647238 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:02.647238 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:22:02.647238 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:02.647238 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:22:02.648109 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:02.648109 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:22:02.648109 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:22:02.648109 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:22:02.648109 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:02.648109 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:22:02.648109 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:02.648109 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:22:02.648109 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:02.648109 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:22:02.648109 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:02.648109 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:22:02.648109 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:02.648109 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:22:02.648109 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:22:02.648553 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:02.648553 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:22:02.648553 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:02.648553 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:22:02.648553 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:22:02.648553 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:02.648553 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:22:02.648553 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:02.648553 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:22:02.649054 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:02.649054 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:22:02.649054 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:02.649054 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:22:02.649054 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:22:02.649054 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:22:02.649054 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:22:02.649054 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 15 10:22:02.649054 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:22:02.649054 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:22:02.649054 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:02.649054 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:02.649449 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:02.649449 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:22:02.649449 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:02.649449 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:02.649449 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:02.649449 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:22:02.649449 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:02.649449 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:22:02.649449 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:22:02.649449 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:02.649449 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:22:02.649449 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:02.649449 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:22:02.649449 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:22:02.649852 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:02.649852 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:22:02.649852 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:02.649852 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:22:02.650157 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:02.650157 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:22:02.650157 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:22:02.650157 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:22:02.650157 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:22:02.650157 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:22:02.650157 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:02.650157 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:22:02.650157 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:22:02.650157 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:02.650157 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:22:02.650157 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:22:02.650157 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:02.650157 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:02.650157 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:22:02.650157 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:02.650651 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:22:02.650651 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:22:02.650651 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^ May 15 10:22:02.650651 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:22:02.650651 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:22:02.650651 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:02.650651 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:02.650651 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:02.650651 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:22:02.650651 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:02.650651 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:02.650651 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:02.650651 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:22:02.650651 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:02.650651 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:22:02.650651 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:22:02.650651 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:02.650651 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:22:02.651222 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:02.651222 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:22:02.651222 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:22:02.651222 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:02.651222 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:22:02.651222 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:02.651222 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:22:02.652157 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:22:02.652468 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_ha_failover {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:22:02.652592 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:22:02.652759 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:22:02.653234 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:22:02.653625 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:02.654482 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:22:02.654482 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:02.654482 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:22:02.654482 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:22:02.654482 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:02.654482 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:22:02.654482 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:02.654482 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:22:02.655394 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:02.655394 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:22:02.655394 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:22:02.655394 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:22:02.655394 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:02.655394 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:22:02.655394 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:02.655394 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:22:02.655394 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:02.655394 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:22:02.655394 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:02.655394 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:22:02.655394 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:02.655394 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:22:02.655394 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:22:02.655848 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:02.655848 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:22:02.655848 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:02.655848 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:22:02.655848 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:22:02.655848 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:02.655848 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:22:02.655848 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:02.655848 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:22:02.656589 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:02.656589 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:22:02.656589 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:02.656589 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:22:02.656589 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:22:02.656589 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:22:02.656589 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 15 10:22:02.656589 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:22:02.656589 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:22:02.656589 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:02.656589 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:02.656926 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:02.656926 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:22:02.656926 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:02.656926 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:02.656926 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:02.656926 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:22:02.656926 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:02.656926 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:22:02.656926 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:22:02.656926 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:02.656926 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:22:02.656926 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:02.656926 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:22:02.656926 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:22:02.657358 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:02.657358 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:22:02.657358 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:02.657358 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:22:02.657874 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:02.657874 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:22:02.657874 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:22:02.657874 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:22:02.657874 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:22:02.657874 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:22:02.657874 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:02.657874 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:22:02.657874 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:22:02.657874 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:02.657874 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:22:02.657874 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:22:02.657874 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:02.657874 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:02.657874 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:22:02.657874 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:02.657874 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:22:02.658355 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 15 10:22:02.658355 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:22:02.658355 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:22:02.658355 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:02.658355 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:02.658355 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:02.658355 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:22:02.658355 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:02.658355 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:02.658355 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:02.658355 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:22:02.658355 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:02.658355 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:22:02.658355 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:22:02.658355 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:02.658355 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:22:02.658355 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:02.658355 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:22:02.658355 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:22:02.659002 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:02.659002 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:22:02.659002 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:02.659002 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:22:02.659002 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:22:02.659002 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:22:02.659002 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:22:02.659002 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:22:02.659422 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:22:02.659881 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:22:02.660101 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:02.660941 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:22:02.660941 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:02.660941 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:22:02.660941 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:22:02.660941 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:02.660941 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:22:02.660941 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:02.660941 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:22:02.661783 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:02.661783 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:22:02.661783 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:02.661783 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:22:02.661783 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:22:02.661783 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:22:02.661783 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:22:02.661783 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:02.661783 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:02.661783 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:02.661783 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:22:02.661783 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:02.662120 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:02.662120 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:02.662120 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:22:02.662120 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:02.662120 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:22:02.662120 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:22:02.662120 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:02.662120 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:22:02.662120 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:02.662120 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:22:02.662120 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:22:02.662120 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:02.662120 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:22:02.662616 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:02.662616 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:22:02.662836 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:02.662836 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:22:02.662836 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:22:02.662836 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:22:02.662836 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:22:02.662836 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:22:02.662836 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:02.662836 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:22:02.662836 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:22:02.662836 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:02.662836 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:22:02.662836 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:22:02.662836 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:02.662836 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:02.662836 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:22:02.662836 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:02.663295 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:22:02.663295 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:22:02.663295 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:02.663295 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:02.663295 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:02.663295 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:22:02.663295 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:02.663295 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:02.663295 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:02.663295 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:22:02.663295 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:02.663295 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:22:02.663295 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:22:02.663295 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:02.663295 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:22:02.663295 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:02.663295 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:22:02.663295 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:22:02.663295 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:02.663839 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:22:02.663839 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:02.663839 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:22:07.525740 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:22:07.526625 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:22:07.526625 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:22:07.527770 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:22:07.555728 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:22:07.555948 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:07.556493 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:22:07.556493 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:07.556493 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:22:07.556493 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:22:07.556493 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:07.556493 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:22:07.556493 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:07.556493 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:22:07.557487 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_fdb_aging_settings failed after 0.030 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:07.557487 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:22:07.557487 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:07.557487 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:22:07.557487 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:22:07.557487 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:22:07.557487 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:22:07.557487 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:07.557487 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:07.557487 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:07.557487 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:22:07.557487 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:07.558220 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:07.558220 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:07.558220 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:22:07.558220 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:07.558220 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:22:07.558220 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:22:07.558220 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:07.558220 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:22:07.558220 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:07.558220 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:22:07.558220 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:22:07.558220 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:07.558220 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:22:07.559143 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:07.559143 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:22:07.559143 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:07.559143 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:22:07.559143 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:22:07.559143 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:22:07.559143 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:22:07.559143 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:22:07.559143 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:07.559143 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:22:07.559143 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:22:07.559143 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:07.559143 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:22:07.559143 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:22:07.559143 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:07.560364 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:07.560364 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:22:07.560364 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:07.560364 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:22:07.560364 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:22:07.560364 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:07.560364 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:07.560364 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:07.560364 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:22:07.560364 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:07.560364 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:07.560364 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:07.560364 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:22:07.560364 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:07.560364 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:22:07.560364 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:22:07.560364 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:07.560364 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:22:07.560364 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:07.561431 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:22:07.561431 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:22:07.561431 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:07.561431 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:22:07.561431 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:07.561431 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:22:07.638356 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:22:07.639156 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:22:07.639449 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:22:07.640346 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b027d63e-13a5-445b-84a0-8ae6223e05d5, routes=[], if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:22:07.640575 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:07.641223 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:22:07.641223 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:07.641223 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:22:07.641223 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:22:07.641223 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:07.641223 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:22:07.641223 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:07.641223 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:22:07.642451 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:07.642451 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:22:07.642451 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:07.642451 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:22:07.642451 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:22:07.642451 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:22:07.642451 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:22:07.642451 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:07.642451 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:07.642451 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:07.642451 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:22:07.642451 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:07.643259 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:07.643259 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:07.643259 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:22:07.643259 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:07.643259 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:22:07.643259 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:22:07.643259 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:07.643259 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:22:07.643259 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:07.643259 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:22:07.643259 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:22:07.643259 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:07.643259 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:22:07.643877 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:07.643877 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:22:07.643877 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:07.643877 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:22:07.643877 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:22:07.643877 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:22:07.643877 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:22:07.643877 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:22:07.643877 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:07.643877 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:22:07.643877 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:22:07.643877 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:07.643877 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:22:07.643877 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:22:07.643877 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:07.644531 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:07.644531 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:22:07.644531 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:07.644531 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:22:07.644531 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:22:07.644531 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:07.644531 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:07.644531 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:07.644531 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:22:07.644531 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:07.644531 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:07.644531 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:07.644531 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:22:07.644531 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:07.644531 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:22:07.644531 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:22:07.644531 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:07.644531 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:22:07.644531 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:07.645332 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:22:07.645332 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:22:07.645332 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:07.645332 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:22:07.645332 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:07.645332 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:22:07.645609 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:22:07.646359 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:22:07.646587 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:22:07.646898 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105458) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 15 10:22:07.647554 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:22:07.647937 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:07.648853 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:22:07.648853 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:07.648853 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:22:07.648853 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:22:07.648853 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:07.648853 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:22:07.648853 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:07.648853 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:22:07.650210 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:07.650210 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:22:07.650210 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:22:07.650210 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:22:07.650210 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:07.650210 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:22:07.650210 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:07.650210 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:22:07.650210 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:07.650210 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:22:07.650210 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:07.650210 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:22:07.650210 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:07.650210 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:22:07.650210 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:22:07.650943 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:07.650943 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:22:07.650943 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:07.650943 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:22:07.650943 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:22:07.650943 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:07.650943 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:22:07.650943 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:07.650943 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:22:07.651708 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:07.651708 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:22:07.651708 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:07.651708 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:22:07.651708 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:22:07.651708 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:22:07.651708 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:22:07.651708 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 15 10:22:07.651708 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:22:07.651708 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:22:07.651708 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:07.651708 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:07.652665 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:07.652665 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:22:07.652665 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:07.652665 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:07.652665 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:07.652665 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:22:07.652665 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:07.652665 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:22:07.652665 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:22:07.652665 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:07.652665 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:22:07.652665 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:07.652665 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:22:07.652665 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:22:07.653548 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:07.653548 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:22:07.653548 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:07.653548 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:22:07.653548 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:07.653548 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:22:07.653548 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:22:07.653548 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:22:07.653548 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:22:07.653548 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:22:07.653548 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:07.653548 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:22:07.653548 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:22:07.653548 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:07.654012 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:22:07.654012 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:22:07.654012 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:07.654012 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:07.654012 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:22:07.654012 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:07.654012 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:22:07.654012 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:22:07.654012 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^ May 15 10:22:07.654012 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:22:07.654012 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:22:07.654012 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:07.654012 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:07.654012 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:07.654012 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:22:07.654012 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:07.654012 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:07.654012 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:07.654012 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:22:07.654628 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:07.654628 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:22:07.654628 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:22:07.654628 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:07.654628 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:22:07.654628 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:07.654628 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:22:07.654628 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:22:07.654628 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:07.654628 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:22:07.654628 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:07.654628 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:22:07.654628 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:22:07.654628 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_ha_failover {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:22:07.655532 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:22:07.655532 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:22:07.655532 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:22:07.655532 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:07.655532 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:22:07.655532 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:07.655532 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:22:07.655532 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:22:07.655532 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:07.655532 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:22:07.655532 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:07.655532 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:22:07.656608 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:07.656608 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:22:07.656608 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:22:07.656608 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:22:07.656608 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:07.656608 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:22:07.656608 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:07.656608 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:22:07.656608 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:07.656608 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:22:07.656608 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:07.656608 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:22:07.656608 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:07.656608 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:22:07.656608 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:22:07.657202 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:07.657202 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:22:07.657202 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:07.657202 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:22:07.657202 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:22:07.657202 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:07.657202 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:22:07.657202 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:07.657202 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:22:07.657698 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:07.657698 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:22:07.657698 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:07.657698 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:22:07.657698 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:22:07.657698 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:22:07.657698 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 15 10:22:07.657698 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:22:07.657698 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:22:07.657698 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:07.657698 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:07.658079 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:07.658079 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:22:07.658079 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:07.658079 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:07.658079 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:07.658079 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:22:07.658079 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:07.658079 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:22:07.658079 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:22:07.658079 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:07.658079 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:22:07.658079 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:07.658079 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:22:07.658079 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:22:07.658472 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:07.658472 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:22:07.658472 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:07.658472 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:22:07.659158 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:07.659158 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:22:07.659158 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:22:07.659158 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:22:07.659158 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:22:07.659158 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:22:07.659158 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:07.659158 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:22:07.659158 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:22:07.659158 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:07.659158 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:22:07.659158 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:22:07.659158 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:07.659158 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:07.659158 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:22:07.659158 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:07.659158 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:22:07.659604 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 15 10:22:07.659604 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:22:07.659604 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:22:07.659604 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:07.659604 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:07.659604 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:07.659604 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:22:07.659604 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:07.659604 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:07.659604 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:07.659604 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:22:07.659604 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:07.659604 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:22:07.659604 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:22:07.659604 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:07.659604 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:22:07.659604 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:07.659604 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:22:07.659604 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:22:07.659981 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:07.659981 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:22:07.659981 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:07.659981 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:22:07.659981 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:22:07.659981 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:22:07.659981 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:22:07.660166 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:22:07.660428 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:22:07.660696 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:22:07.660835 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:07.661439 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:22:07.661439 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:07.661439 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:22:07.661439 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:22:07.661439 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:07.661439 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:22:07.661439 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:07.661439 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:22:07.662428 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:07.662428 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:22:07.662428 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:07.662428 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:22:07.662428 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:22:07.662428 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:22:07.662428 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:22:07.662428 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:07.662428 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:07.662428 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:07.662428 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:22:07.662428 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:07.662729 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:07.662729 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:07.662729 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:22:07.662729 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:07.662729 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:22:07.662729 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:22:07.662729 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:07.662729 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:22:07.662729 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:07.662729 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:22:07.662729 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:22:07.662729 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:07.662729 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:22:07.662974 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:07.662974 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:22:07.663320 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:07.663320 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:22:07.663320 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:22:07.663320 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:22:07.663320 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:22:07.663320 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:22:07.663320 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:07.663320 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:22:07.663320 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:22:07.663320 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:07.663320 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:22:07.663320 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:22:07.663320 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:07.663320 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:07.663320 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:22:07.663320 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:07.663655 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:22:07.663655 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:22:07.663655 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:07.663655 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:07.663655 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:07.663655 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:22:07.663655 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:07.663655 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:07.663655 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:07.663655 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:22:07.663655 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:07.663655 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:22:07.663655 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:22:07.663655 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:07.663655 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:22:07.663655 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:07.663655 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:22:07.663655 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:22:07.663655 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:07.664013 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:22:07.664013 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:07.664013 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:22:12.526250 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:22:12.527448 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:22:12.527448 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:22:12.527964 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:22:12.566131 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:22:12.566399 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:12.567173 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:22:12.567173 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:12.567173 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:22:12.567173 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:22:12.567173 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:12.567173 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:22:12.567173 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:12.567173 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:22:12.568381 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_fdb_aging_settings failed after 0.040 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:12.568381 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:22:12.568381 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:12.568381 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:22:12.568381 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:22:12.568381 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:22:12.568381 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:22:12.568381 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:12.568381 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:12.568381 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:12.568381 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:22:12.568381 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:12.568978 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:12.568978 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:12.568978 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:22:12.568978 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:12.568978 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:22:12.568978 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:22:12.568978 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:12.568978 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:22:12.568978 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:12.568978 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:22:12.568978 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:22:12.568978 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:12.568978 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:22:12.569552 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:12.569552 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:22:12.570765 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:12.570765 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:22:12.570765 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:22:12.570765 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:22:12.570765 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:22:12.570765 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:22:12.570765 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:12.570765 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:22:12.570765 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:22:12.570765 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:12.570765 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:22:12.570765 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:22:12.570765 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:12.570765 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:12.570765 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:22:12.570765 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:12.571505 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:22:12.571505 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:22:12.571505 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:12.571505 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:12.571505 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:12.571505 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:22:12.571505 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:12.571505 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:12.571505 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:12.571505 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:22:12.571505 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:12.571505 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:22:12.571505 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:22:12.571505 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:12.571505 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:22:12.571505 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:12.571505 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:22:12.571505 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:22:12.571505 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:12.572206 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:22:12.572206 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:12.572206 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:22:12.639074 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:22:12.639550 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:22:12.639689 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:22:12.640447 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b027d63e-13a5-445b-84a0-8ae6223e05d5, routes=[], if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:22:12.640744 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:12.641387 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:22:12.641387 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:12.641387 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:22:12.641387 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:22:12.641387 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:12.641387 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:22:12.641387 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:12.641387 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:22:12.642790 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:12.642790 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:22:12.642790 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:12.642790 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:22:12.642790 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:22:12.642790 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:22:12.642790 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:22:12.642790 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:12.642790 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:12.642790 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:12.642790 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:22:12.642790 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:12.643552 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:12.643552 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:12.643552 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:22:12.643552 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:12.643552 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:22:12.643552 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:22:12.643552 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:12.643552 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:22:12.643552 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:12.643552 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:22:12.643552 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:22:12.643552 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:12.643552 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:22:12.644330 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:12.644330 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:22:12.644330 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:12.644330 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:22:12.644330 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:22:12.644330 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:22:12.644330 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:22:12.644330 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:22:12.644330 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:12.644330 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:22:12.644330 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:22:12.644330 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:12.644330 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:22:12.644330 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:22:12.644330 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:12.645076 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:12.645076 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:22:12.645076 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:12.645076 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:22:12.645076 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:22:12.645076 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:12.645076 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:12.645076 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:12.645076 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:22:12.645076 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:12.645076 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:12.645076 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:12.645076 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:22:12.645076 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:12.645076 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:22:12.645076 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:22:12.645076 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:12.645076 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:22:12.645076 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:12.646142 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:22:12.646142 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:22:12.646142 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:12.646142 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:22:12.646142 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:12.646142 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:22:12.646444 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:22:12.646913 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:22:12.647017 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:22:12.647160 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105458) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 15 10:22:12.647710 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:22:12.647936 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:12.648608 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:22:12.648608 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:12.648608 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:22:12.648608 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:22:12.648608 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:12.648608 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:22:12.648608 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:12.648608 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:22:12.650004 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:12.650004 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:22:12.650004 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:22:12.650004 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:22:12.650004 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:12.650004 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:22:12.650004 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:12.650004 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:22:12.650004 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:12.650004 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:22:12.650004 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:12.650004 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:22:12.650004 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:12.650004 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:22:12.650004 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:22:12.650598 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:12.650598 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:22:12.650598 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:12.650598 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:22:12.650598 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:22:12.650598 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:12.650598 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:22:12.650598 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:12.650598 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:22:12.651907 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:12.651907 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:22:12.651907 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:12.651907 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:22:12.651907 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:22:12.651907 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:22:12.651907 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:22:12.651907 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 15 10:22:12.651907 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:22:12.651907 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:22:12.651907 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:12.651907 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:12.652591 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:12.652591 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:22:12.652591 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:12.652591 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:12.652591 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:12.652591 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:22:12.652591 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:12.652591 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:22:12.652591 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:22:12.652591 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:12.652591 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:22:12.652591 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:12.652591 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:22:12.652591 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:22:12.653238 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:12.653238 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:22:12.653238 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:12.653238 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:22:12.653528 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:12.653528 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:22:12.653528 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:22:12.653528 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:22:12.653528 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:22:12.653528 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:22:12.653528 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:12.653528 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:22:12.653528 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:22:12.653528 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:12.653528 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:22:12.653528 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:22:12.653528 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:12.653528 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:12.653528 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:22:12.653528 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:12.654120 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:22:12.654120 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:22:12.654120 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^ May 15 10:22:12.654120 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:22:12.654120 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:22:12.654120 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:12.654120 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:12.654120 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:12.654120 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:22:12.654120 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:12.654120 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:12.654120 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:12.654120 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:22:12.654120 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:12.654120 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:22:12.654120 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:22:12.654120 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:12.654120 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:22:12.654799 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:12.654799 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:22:12.654799 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:22:12.654799 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:12.654799 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:22:12.654799 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:12.654799 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:22:12.654799 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:22:12.654799 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_ha_failover {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:22:12.654799 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:22:12.655188 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:22:12.655501 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:22:12.655921 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:12.656923 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:22:12.656923 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:12.656923 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:22:12.656923 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:22:12.656923 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:12.656923 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:22:12.656923 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:12.656923 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:22:12.657954 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:12.657954 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:22:12.657954 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:22:12.657954 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:22:12.657954 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:12.657954 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:22:12.657954 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:12.657954 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:22:12.657954 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:12.657954 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:22:12.657954 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:12.657954 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:22:12.657954 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:12.657954 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:22:12.657954 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:22:12.658553 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:12.658553 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:22:12.658553 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:12.658553 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:22:12.658553 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:22:12.658553 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:12.658553 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:22:12.658553 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:12.658553 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:22:12.658955 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:12.658955 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:22:12.658955 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:12.658955 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:22:12.658955 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:22:12.658955 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:22:12.658955 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 15 10:22:12.658955 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:22:12.658955 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:22:12.658955 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:12.658955 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:12.659433 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:12.659433 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:22:12.659433 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:12.659433 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:12.659433 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:12.659433 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:22:12.659433 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:12.659433 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:22:12.659433 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:22:12.659433 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:12.659433 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:22:12.659433 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:12.659433 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:22:12.659433 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:22:12.659945 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:12.659945 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:22:12.659945 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:12.659945 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:22:12.660592 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:12.660592 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:22:12.660592 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:22:12.660592 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:22:12.660592 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:22:12.660592 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:22:12.660592 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:12.660592 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:22:12.660592 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:22:12.660592 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:12.660592 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:22:12.660592 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:22:12.660592 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:12.660592 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:12.660592 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:22:12.660592 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:12.660592 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:22:12.661302 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 15 10:22:12.661302 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:22:12.661302 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:22:12.661302 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:12.661302 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:12.661302 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:12.661302 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:22:12.661302 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:12.661302 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:12.661302 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:12.661302 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:22:12.661302 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:12.661302 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:22:12.661302 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:22:12.661302 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:12.661302 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:22:12.661302 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:12.661302 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:22:12.661302 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:22:12.662041 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:12.662041 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:22:12.662041 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:12.662041 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:22:12.662041 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:22:12.662041 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:22:12.662041 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:22:12.662041 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:22:12.662367 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:22:12.662367 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:22:12.662462 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:12.662930 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:22:12.662930 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:12.662930 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:22:12.662930 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:22:12.662930 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:12.662930 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:22:12.662930 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:12.662930 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:22:12.663843 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:12.663843 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:22:12.663843 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:12.663843 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:22:12.663843 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:22:12.663843 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:22:12.663843 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:22:12.663843 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:12.663843 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:12.663843 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:12.663843 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:22:12.663843 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:12.664565 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:12.664565 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:12.664565 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:22:12.664565 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:12.664565 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:22:12.664565 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:22:12.664565 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:12.664565 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:22:12.664565 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:12.664565 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:22:12.664565 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:22:12.664565 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:12.664565 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:22:12.665056 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:12.665056 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:22:12.665056 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:12.665056 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:22:12.665056 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:22:12.665056 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:22:12.665056 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:22:12.665056 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:22:12.665056 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:12.665056 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:22:12.665056 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:22:12.665056 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:12.665056 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:22:12.665056 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:22:12.665056 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:12.665529 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:12.665529 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:22:12.665529 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:12.665529 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:22:12.665529 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:22:12.665529 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:12.665529 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:12.665529 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:12.665529 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:22:12.665529 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:12.665529 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:12.665529 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:12.665529 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:22:12.665529 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:12.665529 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:22:12.665529 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:22:12.665529 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:12.665529 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:22:12.665529 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:12.666211 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:22:12.666211 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:22:12.666211 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:12.666211 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:22:12.666211 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:12.666211 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:22:17.527969 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:22:17.532109 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:22:17.532109 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:22:17.532109 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:22:17.589481 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:22:17.589842 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:17.592584 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:22:17.592584 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:17.592584 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:22:17.592584 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:22:17.592584 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:17.592584 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:22:17.592584 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:17.592584 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:22:17.592584 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_fdb_aging_settings failed after 0.061 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:17.592584 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:22:17.592584 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:17.592584 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:22:17.592584 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:22:17.592584 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:22:17.592584 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:22:17.592917 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:17.592917 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:17.592917 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:17.592917 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:22:17.592917 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:17.592917 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:17.592917 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:17.592917 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:22:17.592917 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:17.592917 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:22:17.592917 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:22:17.592917 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:17.592917 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:22:17.592917 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:17.593214 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:22:17.593214 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:22:17.593214 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:17.593214 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:22:17.593214 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:17.593214 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:22:17.593214 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:17.593214 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:22:17.593214 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:22:17.593214 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:22:17.593214 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:22:17.593214 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:22:17.593214 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:17.593214 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:22:17.593517 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:22:17.593517 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:17.593517 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:22:17.593517 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:22:17.593517 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:17.593517 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:17.593517 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:22:17.593517 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:17.593517 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:22:17.593517 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:22:17.593517 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:17.593517 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:17.593517 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:17.593517 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:22:17.593517 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:17.593517 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:17.593517 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:17.593517 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:22:17.593517 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:17.593517 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:22:17.593517 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:22:17.593980 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:17.593980 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:22:17.593980 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:17.593980 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:22:17.593980 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:22:17.593980 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:17.593980 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:22:17.593980 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:17.593980 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:22:17.639978 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:22:17.641058 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:22:17.641058 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:22:17.641602 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b027d63e-13a5-445b-84a0-8ae6223e05d5, routes=[], if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:22:17.641873 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:17.645744 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:22:17.645744 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:17.645744 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:22:17.645744 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:22:17.645744 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:17.645744 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:22:17.645744 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:17.645744 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:22:17.645744 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:17.645744 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:22:17.645744 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:17.645744 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:22:17.645744 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:22:17.645744 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:22:17.645744 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:22:17.646957 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:17.646957 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:17.646957 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:17.646957 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:22:17.646957 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:17.646957 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:17.646957 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:17.646957 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:22:17.646957 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:17.646957 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:22:17.646957 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:22:17.646957 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:17.646957 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:22:17.646957 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:17.648180 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:22:17.648180 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:22:17.648180 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:17.648180 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:22:17.648180 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:17.648180 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:22:17.650189 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:17.650189 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:22:17.650189 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:22:17.650189 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:22:17.650189 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:22:17.650189 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:22:17.650189 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:17.650189 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:22:17.650189 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:22:17.650189 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:17.650189 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:22:17.650189 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:22:17.650189 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:17.650189 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:17.650189 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:22:17.650189 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:17.651156 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:22:17.651156 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:22:17.651156 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:17.651156 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:17.651156 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:17.651156 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:22:17.651156 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:17.651156 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:17.651156 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:17.651156 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:22:17.651156 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:17.651156 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:22:17.651156 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:22:17.651156 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:17.651156 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:22:17.651156 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:17.651156 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:22:17.651156 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:22:17.651156 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:17.652239 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:22:17.652239 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:17.652239 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:22:17.652903 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:22:17.653916 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:22:17.653916 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:22:17.653916 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105458) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 15 10:22:17.654214 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:22:17.654446 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:17.654963 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:22:17.658585 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:22:17.658585 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:17.658585 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:22:17.658585 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:22:17.658585 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:17.658585 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:22:17.658585 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:17.658585 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:22:17.658585 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:17.658585 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:22:17.658585 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:22:17.658585 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:22:17.658585 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:17.658585 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:22:17.658585 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:17.658585 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:22:17.658585 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:17.658585 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:22:17.659184 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:17.659184 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:22:17.659184 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:17.659184 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:22:17.659184 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:22:17.659184 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:17.659184 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:22:17.659184 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:17.659184 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:22:17.659184 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:22:17.659184 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:17.659184 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:22:17.659184 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:17.659184 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:22:17.659184 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:17.659638 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:22:17.659638 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:17.659638 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:22:17.659638 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:22:17.659638 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:22:17.659638 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:22:17.659638 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 15 10:22:17.659638 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:22:17.659638 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:22:17.659638 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:17.659638 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:17.659638 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:17.659638 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:22:17.660002 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:17.660002 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:17.660002 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:17.660002 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:22:17.660002 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:17.660002 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:22:17.660002 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:22:17.660002 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:17.660002 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:22:17.660002 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:17.660002 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:22:17.660002 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:22:17.660002 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:17.660002 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:22:17.660400 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:17.660400 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:22:17.660400 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:17.660400 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:22:17.660400 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:22:17.660400 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:22:17.660400 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:22:17.660400 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:22:17.660400 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:17.660400 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:22:17.660400 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:22:17.660400 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:17.660400 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:22:17.660400 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:22:17.660400 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:17.660826 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:17.660826 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:22:17.660826 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:17.660826 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:22:17.660826 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:22:17.660826 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^ May 15 10:22:17.660826 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:22:17.660826 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:22:17.660826 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:17.660826 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:17.660826 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:17.660826 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:22:17.660826 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:17.660826 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:17.660826 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:17.660826 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:22:17.660826 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:17.660826 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:22:17.660826 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:22:17.661391 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:17.661391 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:22:17.661391 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:17.661391 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:22:17.661391 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:22:17.661391 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:17.661391 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:22:17.661391 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:17.661391 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:22:17.661391 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_ha_failover {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:22:17.661391 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:22:17.661391 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:22:17.661742 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:22:17.661742 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:17.661742 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:22:17.662092 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:22:17.662092 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:17.662092 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:22:17.662092 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:22:17.662092 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:17.662092 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:22:17.662092 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:17.662092 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:22:17.663142 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:17.663142 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:22:17.663142 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:22:17.663142 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:22:17.663142 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:17.663142 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:22:17.663142 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:17.663142 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:22:17.663142 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:17.663142 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:22:17.663142 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:17.663142 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:22:17.663142 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:17.663142 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:22:17.663142 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:22:17.663604 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:17.663604 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:22:17.663604 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:17.663604 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:22:17.663604 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:22:17.663604 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:17.663604 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:22:17.663604 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:17.663604 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:22:17.664870 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_ha_failover failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:17.664870 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:22:17.664870 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:17.664870 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:22:17.664870 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:22:17.664870 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:22:17.664870 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 15 10:22:17.664870 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:22:17.664870 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:22:17.664870 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:17.664870 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:17.665230 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:17.665230 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:22:17.665230 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:17.665230 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:17.665230 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:17.665230 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:22:17.665230 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:17.665230 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:22:17.665230 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:22:17.665230 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:17.665230 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:22:17.665230 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:17.665230 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:22:17.665230 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:22:17.665619 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:17.665619 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:22:17.665619 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:17.665619 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:22:17.667026 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:17.667026 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:22:17.667026 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:22:17.667026 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:22:17.667026 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:22:17.667026 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:22:17.667026 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:17.667026 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:22:17.667026 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:22:17.667026 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:17.667026 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:22:17.667026 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:22:17.667026 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:17.667026 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:17.667026 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:22:17.667026 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:17.667026 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:22:17.667494 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 15 10:22:17.667494 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:22:17.667494 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:22:17.667494 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:17.667494 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:17.667494 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:17.667494 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:22:17.667494 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:17.667494 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:17.667494 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:17.667494 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:22:17.667494 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:17.667494 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:22:17.667494 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:22:17.667494 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:17.667494 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:22:17.667494 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:17.667494 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:22:17.667494 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:22:17.668004 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:17.668004 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:22:17.668004 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:17.668004 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:22:17.668338 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:22:17.668527 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:22:17.668784 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:22:17.669180 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:22:17.669915 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:22:17.669915 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:17.670171 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:22:17.670171 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:17.670171 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:22:17.670171 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:22:17.670171 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:17.670171 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:22:17.670171 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:17.670171 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:22:17.671455 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:17.671455 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:22:17.671455 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:17.671455 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:22:17.671455 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:22:17.671455 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:22:17.671455 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:22:17.671455 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:17.671455 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:17.671455 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:17.671455 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:22:17.671455 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:17.671835 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:17.671835 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:17.671835 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:22:17.671835 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:17.671835 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:22:17.671835 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:22:17.671835 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:17.671835 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:22:17.671835 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:17.671835 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:22:17.671835 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:22:17.671835 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:17.671835 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:22:17.672215 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:17.672215 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:22:17.673546 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:17.673546 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:22:17.673546 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:22:17.673546 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:22:17.673546 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:22:17.673546 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:22:17.673546 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:17.673546 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:22:17.673546 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:22:17.673546 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:17.673546 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:22:17.673546 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:22:17.673546 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:17.673546 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:17.673546 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:22:17.673546 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:17.674542 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:22:17.674542 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:22:17.674542 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:17.674542 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:17.674542 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:17.674542 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:22:17.674542 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:17.674542 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:17.674542 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:17.674542 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:22:17.674542 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:17.674542 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:22:17.674542 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:22:17.674542 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:17.674542 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:22:17.674542 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:17.674542 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:22:17.674542 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:22:17.674542 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:17.675126 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:22:17.675126 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:17.675126 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:22:22.529731 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:22:22.530790 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:22:22.530790 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:22:22.531136 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:22:22.565346 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:22:22.565778 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:22.566744 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:22:22.566744 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:22.566744 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:22:22.566744 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:22:22.566744 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:22.566744 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:22:22.566744 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:22.566744 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:22:22.567931 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_fdb_aging_settings failed after 0.036 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:22.567931 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:22:22.567931 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:22.567931 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:22:22.567931 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:22:22.567931 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:22:22.567931 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:22:22.567931 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:22.567931 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:22.567931 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:22.567931 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:22:22.567931 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:22.568444 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:22.568444 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:22.568444 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:22:22.568444 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:22.568444 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:22:22.568444 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:22:22.568444 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:22.568444 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:22:22.568444 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:22.568444 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:22:22.568444 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:22:22.568444 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:22.568444 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:22:22.569032 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:22.569032 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:22:22.569400 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:22.569400 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:22:22.569400 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:22:22.569400 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:22:22.569400 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:22:22.569400 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:22:22.569400 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:22.569400 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:22:22.569400 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:22:22.569400 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:22.569400 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:22:22.569400 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:22:22.569400 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:22.569400 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:22.569400 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:22:22.569400 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:22.569909 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:22:22.569909 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:22:22.569909 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:22.569909 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:22.569909 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:22.569909 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:22:22.569909 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:22.569909 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:22.569909 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:22.569909 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:22:22.569909 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:22.569909 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:22:22.569909 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:22:22.569909 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:22.569909 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:22:22.569909 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:22.569909 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:22:22.569909 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:22:22.569909 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:22.570416 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:22:22.570416 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:22.570416 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:22:22.640693 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:22:22.641009 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:22:22.641127 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:22:22.641620 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b027d63e-13a5-445b-84a0-8ae6223e05d5, routes=[], if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:22:22.641812 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:22.642497 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:22:22.642497 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:22.642497 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:22:22.642497 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:22:22.642497 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:22.642497 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:22:22.642497 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:22.642497 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:22:22.644171 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:22.644171 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:22:22.644171 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:22.644171 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:22:22.644171 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:22:22.644171 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:22:22.644171 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:22:22.644171 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:22.644171 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:22.644171 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:22.644171 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:22:22.644171 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:22.644658 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:22.644658 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:22.644658 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:22:22.644658 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:22.644658 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:22:22.644658 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:22:22.644658 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:22.644658 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:22:22.644658 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:22.644658 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:22:22.644658 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:22:22.644658 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:22.644658 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:22:22.645148 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:22.645148 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:22:22.646202 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:22.646202 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:22:22.646202 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:22:22.646202 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:22:22.646202 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:22:22.646202 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:22:22.646202 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:22.646202 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:22:22.646202 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:22:22.646202 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:22.646202 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:22:22.646202 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:22:22.646202 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:22.646202 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:22.646202 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:22:22.646202 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:22.646731 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:22:22.646731 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:22:22.646731 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:22.646731 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:22.646731 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:22.646731 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:22:22.646731 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:22.646731 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:22.646731 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:22.646731 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:22:22.646731 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:22.646731 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:22:22.646731 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:22:22.646731 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:22.646731 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:22:22.646731 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:22.646731 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:22:22.646731 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:22:22.646731 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:22.647248 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:22:22.647248 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:22.647248 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:22:22.653466 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:22:22.653774 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:22:22.653919 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:22:22.654085 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105458) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 15 10:22:22.654835 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:22:22.654835 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:22.655542 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:22:22.655542 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:22.655542 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:22:22.655542 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:22:22.655542 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:22.655542 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:22:22.655542 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:22.655542 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:22:22.656397 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:22.656397 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:22:22.656397 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:22:22.656397 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:22:22.656397 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:22.656397 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:22:22.656397 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:22.656397 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:22:22.656397 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:22.656397 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:22:22.656397 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:22.656397 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:22:22.656397 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:22.656397 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:22:22.656397 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:22:22.656894 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:22.656894 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:22:22.656894 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:22.656894 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:22:22.656894 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:22:22.656894 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:22.656894 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:22:22.656894 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:22.656894 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:22:22.657314 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:22.657314 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:22:22.657314 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:22.657314 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:22:22.657314 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:22:22.657314 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:22:22.657314 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:22:22.657314 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 15 10:22:22.657314 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:22:22.657314 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:22:22.657314 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:22.657314 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:22.657811 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:22.657811 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:22:22.657811 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:22.657811 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:22.657811 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:22.657811 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:22:22.657811 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:22.657811 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:22:22.657811 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:22:22.657811 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:22.657811 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:22:22.657811 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:22.657811 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:22:22.657811 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:22:22.658447 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:22.658447 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:22:22.658447 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:22.658447 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:22:22.658628 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:22.658628 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:22:22.658628 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:22:22.658628 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:22:22.658628 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:22:22.658628 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:22:22.658628 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:22.658628 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:22:22.658628 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:22:22.658628 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:22.658628 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:22:22.658628 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:22:22.658628 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:22.658628 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:22.658628 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:22:22.658628 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:22.659331 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:22:22.659331 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:22:22.659331 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^ May 15 10:22:22.659331 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:22:22.659331 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:22:22.659331 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:22.659331 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:22.659331 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:22.659331 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:22:22.659331 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:22.659331 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:22.659331 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:22.659331 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:22:22.659331 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:22.659331 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:22:22.659331 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:22:22.659331 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:22.659331 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:22:22.660119 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:22.660119 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:22:22.660119 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:22:22.660119 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:22.660119 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:22:22.660119 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:22.660119 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:22:22.660119 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:22:22.660119 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_ha_failover {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:22:22.660119 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:22:22.660580 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:22:22.660580 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:22:22.660580 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:22.660580 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:22:22.660580 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:22.660580 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:22:22.660580 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:22:22.660580 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:22.660580 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:22:22.660580 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:22.660580 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:22:22.661099 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:22.661099 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:22:22.661099 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:22:22.661099 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:22:22.661099 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:22.661099 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:22:22.661099 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:22.661099 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:22:22.661099 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:22.661099 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:22:22.661099 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:22.661099 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:22:22.661099 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:22.661099 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:22:22.661099 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:22:22.661704 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:22.661704 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:22:22.661704 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:22.661704 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:22:22.661704 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:22:22.661704 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:22.661704 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:22:22.661704 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:22.661704 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:22:22.661704 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:22.661704 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:22:22.661704 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:22.661704 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:22:22.661704 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:22:22.662372 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:22:22.662372 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 15 10:22:22.662372 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:22:22.662372 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:22:22.662372 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:22.662372 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:22.662372 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:22.662372 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:22:22.662372 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:22.662372 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:22.662372 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:22.662372 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:22:22.662372 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:22.662932 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:22:22.662932 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:22:22.662932 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:22.662932 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:22:22.662932 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:22.662932 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:22:22.662932 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:22:22.662932 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:22.662932 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:22:22.662932 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:22.662932 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:22:22.662932 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:22.663456 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:22:22.663456 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:22:22.663456 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:22:22.663456 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:22:22.663456 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:22:22.663456 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:22.663456 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:22:22.663456 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:22:22.663456 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:22.663456 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:22:22.663456 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:22:22.663456 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:22.663456 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:22.663456 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:22:22.663456 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:22.663456 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:22:22.663456 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 15 10:22:22.663456 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:22:22.663456 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:22:22.663832 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:22.663832 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:22.663832 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:22.663832 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:22:22.663832 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:22.663832 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:22.663832 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:22.663832 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:22:22.663832 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:22.663832 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:22:22.663832 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:22:22.663832 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:22.663832 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:22:22.663832 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:22.663832 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:22:22.663832 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:22:22.663832 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:22.663832 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:22:22.663832 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:22.663832 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:22:22.668419 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:22:22.668665 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:22:22.668832 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:22:22.669010 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:22:22.669289 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:22:22.669563 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:22:22.669724 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:22.670191 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:22:22.670191 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:22.670191 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:22:22.670191 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:22:22.670191 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:22.670191 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:22:22.670191 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:22.670191 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:22:22.671090 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:22.671090 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:22:22.671090 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:22.671090 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:22:22.671090 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:22:22.671090 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:22:22.671090 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:22:22.671090 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:22.671090 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:22.671090 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:22.671090 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:22:22.671090 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:22.671337 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:22.671337 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:22.671337 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:22:22.671337 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:22.671337 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:22:22.671337 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:22:22.671337 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:22.671337 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:22:22.671337 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:22.671337 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:22:22.671337 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:22:22.671337 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:22.671337 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:22:22.671891 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:22.671891 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:22:22.672420 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:22.672420 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:22:22.672420 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:22:22.672420 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:22:22.672420 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:22:22.672420 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:22:22.672420 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:22.672420 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:22:22.672420 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:22:22.672420 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:22.672420 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:22:22.672420 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:22:22.672420 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:22.672420 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:22.672420 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:22:22.672420 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:22.673176 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:22:22.673176 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:22:22.673176 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:22.673176 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:22.673176 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:22.673176 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:22:22.673176 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:22.673176 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:22.673176 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:22.673176 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:22:22.673176 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:22.673176 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:22:22.673176 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:22:22.673176 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:22.673176 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:22:22.673176 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:22.673176 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:22:22.673176 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:22:22.673176 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:22.674072 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:22:22.674072 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:22.674072 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:22:27.530352 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:22:27.531128 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:22:27.531128 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:22:27.532319 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:22:27.568282 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:22:27.568608 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:27.569478 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:22:27.569478 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:27.569478 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:22:27.569478 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:22:27.569478 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:27.569478 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:22:27.569478 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:27.569478 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:22:27.570529 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_fdb_aging_settings failed after 0.038 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:27.570529 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:22:27.570529 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:27.570529 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:22:27.570529 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:22:27.570529 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:22:27.570529 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:22:27.570529 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:27.570529 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:27.570529 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:27.570529 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:22:27.570529 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:27.570996 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:27.570996 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:27.570996 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:22:27.570996 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:27.570996 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:22:27.570996 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:22:27.570996 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:27.570996 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:22:27.570996 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:27.570996 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:22:27.570996 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:22:27.570996 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:27.570996 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:22:27.571346 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:27.571346 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:22:27.571492 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:27.571492 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:22:27.571492 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:22:27.571492 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:22:27.571492 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:22:27.571492 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:22:27.571492 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:27.571492 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:22:27.571492 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:22:27.571492 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:27.571492 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:22:27.571492 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:22:27.571492 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:27.571492 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:27.571492 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:22:27.571492 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:27.572031 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:22:27.572031 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:22:27.572031 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:27.572031 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:27.572031 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:27.572031 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:22:27.572031 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:27.572031 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:27.572031 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:27.572031 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:22:27.572031 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:27.572031 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:22:27.572031 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:22:27.572031 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:27.572031 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:22:27.572031 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:27.572031 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:22:27.572031 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:22:27.572031 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:27.572512 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:22:27.572512 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:27.572512 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:22:27.641486 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:22:27.641887 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:22:27.642112 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:22:27.643043 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b027d63e-13a5-445b-84a0-8ae6223e05d5, routes=[], if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:22:27.643314 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:27.644199 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:22:27.644199 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:27.644199 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:22:27.644199 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:22:27.644199 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:27.644199 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:22:27.644199 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:27.644199 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:22:27.645701 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:27.645701 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:22:27.645701 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:27.645701 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:22:27.645701 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:22:27.645701 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:22:27.645701 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:22:27.645701 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:27.645701 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:27.645701 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:27.645701 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:22:27.645701 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:27.646325 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:27.646325 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:27.646325 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:22:27.646325 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:27.646325 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:22:27.646325 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:22:27.646325 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:27.646325 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:22:27.646325 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:27.646325 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:22:27.646325 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:22:27.646325 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:27.646325 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:22:27.647289 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:27.647289 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:22:27.647701 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:27.647701 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:22:27.647701 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:22:27.647701 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:22:27.647701 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:22:27.647701 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:22:27.647701 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:27.647701 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:22:27.647701 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:22:27.647701 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:27.647701 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:22:27.647701 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:22:27.647701 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:27.647701 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:27.647701 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:22:27.647701 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:27.648376 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:22:27.648376 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:22:27.648376 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:27.648376 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:27.648376 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:27.648376 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:22:27.648376 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:27.648376 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:27.648376 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:27.648376 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:22:27.648376 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:27.648376 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:22:27.648376 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:22:27.648376 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:27.648376 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:22:27.648376 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:27.648376 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:22:27.648376 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:22:27.648376 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:27.649426 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:22:27.649426 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:27.649426 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:22:27.653909 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:22:27.654730 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:22:27.654730 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:22:27.654730 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105458) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 15 10:22:27.655448 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:22:27.655790 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:27.656507 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:22:27.656507 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:27.656507 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:22:27.656507 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:22:27.656507 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:27.656507 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:22:27.656507 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:27.656507 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:22:27.657303 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:27.657303 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:22:27.657303 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:22:27.657303 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:22:27.657303 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:27.657303 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:22:27.657303 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:27.657303 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:22:27.657303 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:27.657303 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:22:27.657303 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:27.657303 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:22:27.657303 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:27.657303 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:22:27.657303 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:22:27.657817 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:27.657817 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:22:27.657817 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:27.657817 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:22:27.657817 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:22:27.657817 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:27.657817 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:22:27.657817 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:27.657817 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:22:27.658186 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:27.658186 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:22:27.658186 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:27.658186 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:22:27.658186 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:22:27.658186 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:22:27.658186 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:22:27.658186 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 15 10:22:27.658186 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:22:27.658186 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:22:27.658186 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:27.658186 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:27.658565 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:27.658565 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:22:27.658565 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:27.658565 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:27.658565 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:27.658565 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:22:27.658565 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:27.658565 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:22:27.658565 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:22:27.658565 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:27.658565 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:22:27.658565 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:27.658565 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:22:27.658565 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:22:27.659023 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:27.659023 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:22:27.659023 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:27.659023 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:22:27.659332 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:27.659332 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:22:27.659332 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:22:27.659332 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:22:27.659332 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:22:27.659332 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:22:27.659332 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:27.659332 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:22:27.659332 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:22:27.659332 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:27.659332 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:22:27.659332 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:22:27.659332 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:27.659332 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:27.659332 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:22:27.659332 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:27.659855 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:22:27.659855 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:22:27.659855 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^ May 15 10:22:27.659855 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:22:27.659855 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:22:27.659855 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:27.659855 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:27.659855 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:27.659855 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:22:27.659855 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:27.659855 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:27.659855 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:27.659855 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:22:27.659855 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:27.659855 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:22:27.659855 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:22:27.659855 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:27.659855 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:22:27.660392 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:27.660392 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:22:27.660392 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:22:27.660392 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:27.660392 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:22:27.660392 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:27.660392 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:22:27.660392 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:22:27.660392 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_ha_failover {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:22:27.660392 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:22:27.660750 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:22:27.660750 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:22:27.660868 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:27.661482 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:22:27.661482 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:27.661482 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:22:27.661482 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:22:27.661482 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:27.661482 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:22:27.661482 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:27.661482 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:22:27.662241 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:27.662241 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:22:27.662241 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:22:27.662241 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:22:27.662241 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:27.662241 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:22:27.662241 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:27.662241 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:22:27.662241 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:27.662241 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:22:27.662241 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:27.662241 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:22:27.662241 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:27.662241 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:22:27.662241 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:22:27.662755 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:27.662755 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:22:27.662755 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:27.662755 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:22:27.662755 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:22:27.662755 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:27.662755 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:22:27.662755 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:27.662755 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:22:27.663243 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:27.663243 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:22:27.663243 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:27.663243 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:22:27.663243 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:22:27.663243 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:22:27.663243 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 15 10:22:27.663243 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:22:27.663243 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:22:27.663243 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:27.663243 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:27.663611 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:27.663611 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:22:27.663611 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:27.663611 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:27.663611 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:27.663611 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:22:27.663611 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:27.663611 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:22:27.663611 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:22:27.663611 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:27.663611 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:22:27.663611 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:27.663611 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:22:27.663611 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:22:27.664079 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:27.664079 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:22:27.664079 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:27.664079 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:22:27.664497 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:27.664497 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:22:27.664497 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:22:27.664497 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:22:27.664497 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:22:27.664497 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:22:27.664497 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:27.664497 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:22:27.664497 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:22:27.664497 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:27.664497 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:22:27.664497 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:22:27.664497 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:27.664497 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:27.664497 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:22:27.664497 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:27.664497 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:22:27.665109 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 15 10:22:27.665109 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:22:27.665109 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:22:27.665109 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:27.665109 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:27.665109 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:27.665109 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:22:27.665109 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:27.665109 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:27.665109 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:27.665109 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:22:27.665109 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:27.665109 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:22:27.665109 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:22:27.665109 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:27.665109 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:22:27.665109 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:27.665109 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:22:27.665109 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:22:27.666277 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:27.666277 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:22:27.666277 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:27.666277 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:22:27.668865 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:22:27.669122 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:22:27.669271 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:22:27.669412 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:22:27.669824 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:22:27.670217 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:22:27.670449 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:27.671271 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:22:27.671271 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:27.671271 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:22:27.671271 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:22:27.671271 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:27.671271 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:22:27.671271 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:27.671271 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:22:27.672565 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:27.672565 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:22:27.672565 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:27.672565 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:22:27.672565 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:22:27.672565 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:22:27.672565 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:22:27.672565 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:27.672565 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:27.672565 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:27.672565 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:22:27.672565 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:27.672993 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:27.672993 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:27.672993 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:22:27.672993 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:27.672993 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:22:27.672993 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:22:27.672993 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:27.672993 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:22:27.672993 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:27.672993 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:22:27.672993 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:22:27.672993 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:27.672993 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:22:27.673378 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:27.673378 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:22:27.673775 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:27.673775 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:22:27.673775 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:22:27.673775 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:22:27.673775 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:22:27.673775 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:22:27.673775 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:27.673775 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:22:27.673775 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:22:27.673775 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:27.673775 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:22:27.673775 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:22:27.673775 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:27.673775 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:27.673775 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:22:27.673775 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:27.674311 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:22:27.674311 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:22:27.674311 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:27.674311 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:27.674311 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:27.674311 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:22:27.674311 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:27.674311 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:27.674311 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:27.674311 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:22:27.674311 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:27.674311 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:22:27.674311 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:22:27.674311 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:27.674311 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:22:27.674311 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:27.674311 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:22:27.674311 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:22:27.674311 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:27.674982 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:22:27.674982 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:27.674982 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:22:32.531281 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:22:32.532907 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:22:32.533525 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:22:32.534712 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:22:32.563699 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:22:32.564151 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:32.564965 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:22:32.564965 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:32.564965 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:22:32.564965 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:22:32.564965 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:32.564965 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:22:32.564965 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:32.564965 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:22:32.565726 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_fdb_aging_settings failed after 0.032 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:32.565726 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:22:32.565726 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:32.565726 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:22:32.565726 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:22:32.565726 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:22:32.565726 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:22:32.565726 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:32.565726 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:32.565726 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:32.565726 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:22:32.565726 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:32.566286 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:32.566286 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:32.566286 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:22:32.566286 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:32.566286 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:22:32.566286 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:22:32.566286 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:32.566286 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:22:32.566286 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:32.566286 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:22:32.566286 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:22:32.566286 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:32.566286 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:22:32.566892 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:32.566892 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:22:32.566892 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:32.566892 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:22:32.566892 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:22:32.566892 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:22:32.566892 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:22:32.566892 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:22:32.566892 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:32.566892 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:22:32.566892 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:22:32.566892 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:32.566892 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:22:32.566892 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:22:32.566892 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:32.567550 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:32.567550 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:22:32.567550 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:32.567550 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:22:32.567550 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:22:32.567550 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:32.567550 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:32.567550 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:32.567550 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:22:32.567550 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:32.567550 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:32.567550 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:32.567550 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:22:32.567550 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:32.567550 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:22:32.567550 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:22:32.567550 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:32.567550 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:22:32.567550 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:32.568579 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:22:32.568579 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:22:32.568579 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:32.568579 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:22:32.568579 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:32.568579 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:22:32.642025 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:22:32.642624 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:22:32.642848 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:22:32.643600 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b027d63e-13a5-445b-84a0-8ae6223e05d5, routes=[], if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:22:32.644017 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:32.644875 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:22:32.644875 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:32.644875 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:22:32.644875 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:22:32.644875 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:32.644875 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:22:32.644875 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:32.644875 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:22:32.646461 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:32.646461 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:22:32.646461 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:32.646461 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:22:32.646461 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:22:32.646461 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:22:32.646461 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:22:32.646461 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:32.646461 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:32.646461 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:32.646461 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:22:32.646461 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:32.647086 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:32.647086 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:32.647086 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:22:32.647086 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:32.647086 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:22:32.647086 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:22:32.647086 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:32.647086 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:22:32.647086 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:32.647086 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:22:32.647086 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:22:32.647086 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:32.647086 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:22:32.647442 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:32.647442 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:22:32.648365 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:32.648365 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:22:32.648365 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:22:32.648365 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:22:32.648365 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:22:32.648365 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:22:32.648365 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:32.648365 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:22:32.648365 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:22:32.648365 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:32.648365 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:22:32.648365 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:22:32.648365 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:32.648365 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:32.648365 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:22:32.648365 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:32.648941 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:22:32.648941 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:22:32.648941 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:32.648941 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:32.648941 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:32.648941 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:22:32.648941 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:32.648941 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:32.648941 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:32.648941 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:22:32.648941 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:32.648941 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:22:32.648941 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:22:32.648941 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:32.648941 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:22:32.648941 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:32.648941 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:22:32.648941 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:22:32.648941 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:32.649497 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:22:32.649497 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:32.649497 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:22:32.654345 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:22:32.654753 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:22:32.654891 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:22:32.655020 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105458) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 15 10:22:32.655418 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:22:32.655656 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:32.656618 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:22:32.656618 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:32.656618 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:22:32.656618 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:22:32.656618 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:32.656618 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:22:32.656618 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:32.656618 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:22:32.657533 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:32.657533 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:22:32.657533 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:22:32.657533 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:22:32.657533 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:32.657533 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:22:32.657533 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:32.657533 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:22:32.657533 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:32.657533 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:22:32.657533 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:32.657533 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:22:32.657533 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:32.657533 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:22:32.657533 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:22:32.658189 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:32.658189 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:22:32.658189 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:32.658189 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:22:32.658189 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:22:32.658189 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:32.658189 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:22:32.658189 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:32.658189 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:22:32.658538 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:32.658538 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:22:32.658538 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:32.658538 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:22:32.658538 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:22:32.658538 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:22:32.658538 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:22:32.658538 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 15 10:22:32.658538 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:22:32.658538 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:22:32.658538 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:32.658538 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:32.659129 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:32.659129 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:22:32.659129 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:32.659129 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:32.659129 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:32.659129 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:22:32.659129 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:32.659129 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:22:32.659129 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:22:32.659129 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:32.659129 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:22:32.659129 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:32.659129 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:22:32.659129 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:22:32.659783 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:32.659783 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:22:32.659783 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:32.659783 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:22:32.660126 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:32.660126 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:22:32.660126 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:22:32.660126 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:22:32.660126 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:22:32.660126 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:22:32.660126 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:32.660126 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:22:32.660126 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:22:32.660126 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:32.660126 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:22:32.660126 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:22:32.660126 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:32.660126 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:32.660126 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:22:32.660126 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:32.660875 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:22:32.660875 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:22:32.660875 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^ May 15 10:22:32.660875 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:22:32.660875 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:22:32.660875 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:32.660875 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:32.660875 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:32.660875 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:22:32.660875 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:32.660875 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:32.660875 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:32.660875 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:22:32.660875 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:32.660875 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:22:32.660875 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:22:32.660875 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:32.660875 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:22:32.661341 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:32.661341 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:22:32.661341 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:22:32.661341 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:32.661341 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:22:32.661341 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:32.661341 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:22:32.661341 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:22:32.661341 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_ha_failover {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:22:32.661341 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:22:32.661615 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:22:32.661615 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:22:32.661940 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:32.662763 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:22:32.662763 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:32.662763 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:22:32.662763 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:22:32.662763 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:32.662763 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:22:32.662763 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:32.662763 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:22:32.663608 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:32.663608 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:22:32.663608 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:22:32.663608 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:22:32.663608 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:32.663608 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:22:32.663608 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:32.663608 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:22:32.663608 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:32.663608 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:22:32.663608 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:32.663608 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:22:32.663608 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:32.663608 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:22:32.663608 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:22:32.664089 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:32.664089 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:22:32.664089 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:32.664089 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:22:32.664089 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:22:32.664089 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:32.664089 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:22:32.664089 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:32.664089 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:22:32.664623 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:32.664623 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:22:32.664623 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:32.664623 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:22:32.664623 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:22:32.664623 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:22:32.664623 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 15 10:22:32.664623 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:22:32.664623 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:22:32.664623 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:32.664623 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:32.664991 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:32.664991 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:22:32.664991 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:32.664991 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:32.664991 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:32.664991 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:22:32.664991 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:32.664991 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:22:32.664991 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:22:32.664991 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:32.664991 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:22:32.664991 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:32.664991 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:22:32.664991 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:22:32.665351 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:32.665351 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:22:32.665351 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:32.665351 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:22:32.665930 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:32.665930 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:22:32.665930 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:22:32.665930 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:22:32.665930 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:22:32.665930 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:22:32.665930 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:32.665930 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:22:32.665930 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:22:32.665930 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:32.665930 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:22:32.665930 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:22:32.665930 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:32.665930 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:32.665930 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:22:32.665930 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:32.665930 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:22:32.666331 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 15 10:22:32.666331 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:22:32.666331 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:22:32.666331 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:32.666331 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:32.666331 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:32.666331 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:22:32.666331 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:32.666331 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:32.666331 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:32.666331 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:22:32.666331 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:32.666331 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:22:32.666331 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:22:32.666331 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:32.666331 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:22:32.666331 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:32.666331 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:22:32.666331 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:22:32.666803 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:32.666803 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:22:32.666803 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:32.666803 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:22:32.669278 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:22:32.669588 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:22:32.669762 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:22:32.669970 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:22:32.670630 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:22:32.670893 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:22:32.671016 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:32.671595 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:22:32.671595 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:32.671595 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:22:32.671595 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:22:32.671595 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:32.671595 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:22:32.671595 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:32.671595 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:22:32.672754 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:32.672754 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:22:32.672754 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:32.672754 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:22:32.672754 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:22:32.672754 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:22:32.672754 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:22:32.672754 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:32.672754 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:32.672754 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:32.672754 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:22:32.672754 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:32.673119 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:32.673119 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:32.673119 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:22:32.673119 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:32.673119 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:22:32.673119 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:22:32.673119 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:32.673119 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:22:32.673119 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:32.673119 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:22:32.673119 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:22:32.673119 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:32.673119 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:22:32.673525 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:32.673525 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:22:32.673882 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:32.673882 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:22:32.673882 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:22:32.673882 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:22:32.673882 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:22:32.673882 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:22:32.673882 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:32.673882 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:22:32.673882 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:22:32.673882 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:32.673882 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:22:32.673882 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:22:32.673882 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:32.673882 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:32.673882 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:22:32.673882 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:32.674608 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:22:32.674608 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:22:32.674608 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:32.674608 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:32.674608 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:32.674608 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:22:32.674608 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:32.674608 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:32.674608 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:32.674608 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:22:32.674608 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:32.674608 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:22:32.674608 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:22:32.674608 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:32.674608 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:22:32.674608 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:32.674608 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:22:32.674608 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:22:32.674608 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:32.675587 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:22:32.675587 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:32.675587 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:22:37.532433 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:22:37.535078 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:22:37.535206 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:22:37.536698 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:22:37.569879 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:22:37.570148 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:37.570803 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:22:37.570803 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:37.570803 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:22:37.570803 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:22:37.570803 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:37.570803 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:22:37.570803 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:37.570803 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:22:37.571504 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_fdb_aging_settings failed after 0.036 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:37.571504 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:22:37.571504 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:37.571504 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:22:37.571504 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:22:37.571504 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:22:37.571504 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:22:37.571504 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:37.571504 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:37.571504 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:37.571504 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:22:37.571504 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:37.571985 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:37.571985 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:37.571985 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:22:37.571985 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:37.571985 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:22:37.571985 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:22:37.571985 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:37.571985 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:22:37.571985 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:37.571985 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:22:37.571985 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:22:37.571985 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:37.571985 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:22:37.572365 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:37.572365 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:22:37.572365 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:37.572365 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:22:37.572365 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:22:37.572365 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:22:37.572365 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:22:37.572365 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:22:37.572365 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:37.572365 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:22:37.572365 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:22:37.572365 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:37.572365 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:22:37.572365 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:22:37.572365 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:37.572971 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:37.572971 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:22:37.572971 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:37.572971 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:22:37.572971 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:22:37.572971 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:37.572971 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:37.572971 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:37.572971 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:22:37.572971 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:37.572971 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:37.572971 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:37.572971 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:22:37.572971 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:37.572971 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:22:37.572971 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:22:37.572971 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:37.572971 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:22:37.572971 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:37.573529 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:22:37.573529 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:22:37.573529 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:37.573529 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:22:37.573529 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:37.573529 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:22:37.642710 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:22:37.643315 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:22:37.643504 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:22:37.644374 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b027d63e-13a5-445b-84a0-8ae6223e05d5, routes=[], if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:22:37.644650 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:37.645543 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:22:37.645543 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:37.645543 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:22:37.645543 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:22:37.645543 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:37.645543 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:22:37.645543 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:37.645543 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:22:37.646613 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:37.646613 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:22:37.646613 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:37.646613 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:22:37.646613 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:22:37.646613 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:22:37.646613 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:22:37.646613 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:37.646613 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:37.646613 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:37.646613 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:22:37.646613 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:37.647194 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:37.647194 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:37.647194 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:22:37.647194 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:37.647194 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:22:37.647194 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:22:37.647194 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:37.647194 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:22:37.647194 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:37.647194 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:22:37.647194 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:22:37.647194 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:37.647194 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:22:37.647725 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:37.647725 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:22:37.647791 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:37.647791 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:22:37.647791 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:22:37.647791 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:22:37.647791 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:22:37.647791 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:22:37.647791 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:37.647791 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:22:37.647791 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:22:37.647791 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:37.647791 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:22:37.647791 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:22:37.647791 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:37.647791 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:37.647791 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:22:37.647791 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:37.648183 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:22:37.648183 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:22:37.648183 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:37.648183 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:37.648183 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:37.648183 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:22:37.648183 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:37.648183 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:37.648183 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:37.648183 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:22:37.648183 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:37.648183 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:22:37.648183 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:22:37.648183 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:37.648183 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:22:37.648183 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:37.648183 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:22:37.648183 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:22:37.648183 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:37.649009 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:22:37.649009 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:37.649009 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:22:37.654764 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:22:37.655093 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:22:37.655245 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:22:37.655450 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105458) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 15 10:22:37.655809 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:22:37.656028 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:37.656739 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:22:37.656739 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:37.656739 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:22:37.656739 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:22:37.656739 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:37.656739 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:22:37.656739 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:37.656739 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:22:37.657838 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:37.657838 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:22:37.657838 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:22:37.657838 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:22:37.657838 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:37.657838 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:22:37.657838 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:37.657838 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:22:37.657838 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:37.657838 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:22:37.657838 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:37.657838 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:22:37.657838 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:37.657838 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:22:37.657838 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:22:37.658518 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:37.658518 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:22:37.658518 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:37.658518 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:22:37.658518 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:22:37.658518 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:37.658518 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:22:37.658518 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:37.658518 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:22:37.659624 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:37.659624 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:22:37.659624 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:37.659624 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:22:37.659624 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:22:37.659624 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:22:37.659624 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:22:37.659624 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 15 10:22:37.659624 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:22:37.659624 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:22:37.659624 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:37.659624 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:37.660621 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:37.660621 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:22:37.660621 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:37.660621 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:37.660621 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:37.660621 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:22:37.660621 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:37.660621 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:22:37.660621 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:22:37.660621 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:37.660621 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:22:37.660621 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:37.660621 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:22:37.660621 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:22:37.661465 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:37.661465 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:22:37.661465 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:37.661465 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:22:37.661465 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:37.661465 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:22:37.661465 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:22:37.661465 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:22:37.661465 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:22:37.661465 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:22:37.661465 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:37.661465 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:22:37.661465 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:22:37.661465 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:37.662349 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:22:37.662349 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:22:37.662349 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:37.662349 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:37.662349 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:22:37.662349 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:37.662349 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:22:37.662349 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:22:37.662349 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^ May 15 10:22:37.662349 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:22:37.662349 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:22:37.662349 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:37.662349 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:37.662349 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:37.662349 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:22:37.662349 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:37.662349 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:37.662349 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:37.662349 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:22:37.663147 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:37.663147 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:22:37.663147 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:22:37.663147 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:37.663147 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:22:37.663147 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:37.663147 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:22:37.663147 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:22:37.663147 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:37.663147 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:22:37.663147 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:37.663147 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:22:37.663147 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:22:37.663147 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_ha_failover {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:22:37.663893 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:22:37.663893 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:22:37.663893 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:22:37.663893 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:37.663893 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:22:37.663893 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:37.663893 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:22:37.663893 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:22:37.663893 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:37.663893 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:22:37.663893 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:37.663893 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:22:37.665092 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:37.665092 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:22:37.665092 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:22:37.665092 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:22:37.665092 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:37.665092 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:22:37.665092 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:37.665092 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:22:37.665092 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:37.665092 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:22:37.665092 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:37.665092 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:22:37.665092 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:37.665092 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:22:37.665092 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:22:37.666059 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:37.666059 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:22:37.666059 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:37.666059 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:22:37.666059 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:22:37.666059 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:37.666059 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:22:37.666059 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:37.666059 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:22:37.666831 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:37.666831 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:22:37.666831 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:37.666831 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:22:37.666831 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:22:37.666831 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:22:37.666831 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 15 10:22:37.666831 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:22:37.666831 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:22:37.666831 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:37.666831 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:37.667649 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:37.667649 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:22:37.667649 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:37.667649 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:37.667649 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:37.667649 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:22:37.667649 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:37.667649 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:22:37.667649 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:22:37.667649 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:37.667649 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:22:37.667649 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:37.667649 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:22:37.667649 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:22:37.668456 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:37.668456 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:22:37.668456 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:37.668456 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:22:37.668456 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:37.668456 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:22:37.668456 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:22:37.668456 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:22:37.668456 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:22:37.668456 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:22:37.668456 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:37.668456 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:22:37.668456 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:22:37.668456 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:37.669131 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:22:37.669131 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:22:37.669131 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:37.669131 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:37.669131 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:22:37.669131 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:37.669131 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:22:37.669131 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 15 10:22:37.669131 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:22:37.669131 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:22:37.669131 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:37.669131 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:37.669131 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:37.669131 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:22:37.669131 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:37.669131 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:37.669131 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:37.669131 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:22:37.669131 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:37.670022 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:22:37.670022 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:22:37.670022 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:37.670022 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:22:37.670022 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:37.670022 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:22:37.670022 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:22:37.670022 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:37.670022 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:22:37.670022 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:37.670022 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:22:37.670022 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:22:37.670022 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:22:37.670974 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:22:37.670974 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:22:37.670974 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:22:37.671358 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:22:37.671358 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:37.671992 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:22:37.671992 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:37.671992 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:22:37.671992 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:22:37.671992 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:37.671992 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:22:37.671992 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:37.671992 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:22:37.673152 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:37.673152 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:22:37.673152 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:37.673152 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:22:37.673152 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:22:37.673152 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:22:37.673152 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:22:37.673152 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:37.673152 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:37.673152 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:37.673152 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:22:37.673152 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:37.673514 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:37.673514 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:37.673514 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:22:37.673514 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:37.673514 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:22:37.673514 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:22:37.673514 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:37.673514 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:22:37.673514 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:37.673514 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:22:37.673514 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:22:37.673514 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:37.673514 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:22:37.674175 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:37.674175 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:22:37.674175 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:37.674175 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:22:37.674175 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:22:37.674175 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:22:37.674175 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:22:37.674175 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:22:37.674175 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:37.674175 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:22:37.674175 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:22:37.674175 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:37.674175 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:22:37.674175 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:22:37.674175 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:37.674639 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:37.674639 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:22:37.674639 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:37.674639 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:22:37.674639 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:22:37.674639 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:37.674639 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:37.674639 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:37.674639 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:22:37.674639 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:37.674639 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:37.674639 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:37.674639 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:22:37.674639 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:37.674639 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:22:37.674639 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:22:37.674639 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:37.674639 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:22:37.674639 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:37.675139 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:22:37.675139 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:22:37.675139 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:37.675139 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:22:37.675139 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:37.675139 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:22:42.535352 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:22:42.536050 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:22:42.536209 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:22:42.537314 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:22:42.579713 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:22:42.580074 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:42.580888 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:22:42.580888 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:42.580888 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:22:42.580888 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:22:42.580888 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:42.580888 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:22:42.580888 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:42.580888 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:22:42.581748 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_fdb_aging_settings failed after 0.045 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:42.581748 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:22:42.581748 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:42.581748 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:22:42.581748 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:22:42.581748 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:22:42.581748 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:22:42.581748 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:42.581748 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:42.581748 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:42.581748 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:22:42.581748 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:42.582400 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:42.582400 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:42.582400 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:22:42.582400 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:42.582400 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:22:42.582400 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:22:42.582400 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:42.582400 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:22:42.582400 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:42.582400 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:22:42.582400 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:22:42.582400 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:42.582400 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:22:42.582799 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:42.582799 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:22:42.582799 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:42.582799 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:22:42.582799 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:22:42.582799 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:22:42.582799 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:22:42.582799 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:22:42.582799 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:42.582799 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:22:42.582799 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:22:42.582799 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:42.582799 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:22:42.582799 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:22:42.582799 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:42.583280 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:42.583280 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:22:42.583280 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:42.583280 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:22:42.583280 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:22:42.583280 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:42.583280 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:42.583280 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:42.583280 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:22:42.583280 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:42.583280 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:42.583280 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:42.583280 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:22:42.583280 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:42.583280 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:22:42.583280 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:22:42.583280 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:42.583280 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:22:42.583280 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:42.583805 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:22:42.583805 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:22:42.583805 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:42.583805 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:22:42.583805 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:42.583805 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:22:42.643473 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:22:42.644005 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:22:42.644197 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:22:42.645181 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b027d63e-13a5-445b-84a0-8ae6223e05d5, routes=[], if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:22:42.645371 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:42.646114 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:22:42.646114 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:42.646114 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:22:42.646114 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:22:42.646114 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:42.646114 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:22:42.646114 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:42.646114 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:22:42.647835 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:42.647835 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:22:42.647835 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:42.647835 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:22:42.647835 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:22:42.647835 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:22:42.647835 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:22:42.647835 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:42.647835 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:42.647835 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:42.647835 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:22:42.647835 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:42.648446 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:42.648446 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:42.648446 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:22:42.648446 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:42.648446 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:22:42.648446 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:22:42.648446 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:42.648446 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:22:42.648446 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:42.648446 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:22:42.648446 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:22:42.648446 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:42.648446 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:22:42.649023 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:42.649023 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:22:42.649977 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:42.649977 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:22:42.649977 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:22:42.649977 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:22:42.649977 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:22:42.649977 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:22:42.649977 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:42.649977 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:22:42.649977 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:22:42.649977 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:42.649977 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:22:42.649977 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:22:42.649977 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:42.649977 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:42.649977 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:22:42.649977 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:42.650910 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:22:42.650910 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:22:42.650910 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:42.650910 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:42.650910 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:42.650910 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:22:42.650910 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:42.650910 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:42.650910 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:42.650910 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:22:42.650910 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:42.650910 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:22:42.650910 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:22:42.650910 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:42.650910 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:22:42.650910 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:42.650910 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:22:42.650910 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:22:42.650910 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:42.651791 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:22:42.651791 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:42.651791 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:22:42.655224 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:22:42.655489 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:22:42.655608 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:22:42.655747 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105458) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 15 10:22:42.656258 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:22:42.656632 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:42.657394 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:22:42.657394 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:42.657394 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:22:42.657394 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:22:42.657394 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:42.657394 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:22:42.657394 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:42.657394 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:22:42.658127 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:42.658127 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:22:42.658127 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:22:42.658127 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:22:42.658127 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:42.658127 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:22:42.658127 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:42.658127 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:22:42.658127 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:42.658127 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:22:42.658127 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:42.658127 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:22:42.658127 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:42.658127 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:22:42.658127 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:22:42.658553 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:42.658553 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:22:42.658553 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:42.658553 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:22:42.658553 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:22:42.658553 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:42.658553 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:22:42.658553 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:42.658553 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:22:42.659101 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:42.659101 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:22:42.659101 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:42.659101 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:22:42.659101 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:22:42.659101 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:22:42.659101 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:22:42.659101 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 15 10:22:42.659101 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:22:42.659101 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:22:42.659101 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:42.659101 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:42.659469 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:42.659469 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:22:42.659469 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:42.659469 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:42.659469 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:42.659469 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:22:42.659469 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:42.659469 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:22:42.659469 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:22:42.659469 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:42.659469 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:22:42.659469 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:42.659469 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:22:42.659469 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:22:42.659878 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:42.659878 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:22:42.659878 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:42.659878 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:22:42.660306 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:42.660306 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:22:42.660306 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:22:42.660306 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:22:42.660306 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:22:42.660306 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:22:42.660306 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:42.660306 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:22:42.660306 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:22:42.660306 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:42.660306 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:22:42.660306 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:22:42.660306 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:42.660306 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:42.660306 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:22:42.660306 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:42.660787 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:22:42.660787 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:22:42.660787 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^ May 15 10:22:42.660787 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:22:42.660787 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:22:42.660787 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:42.660787 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:42.660787 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:42.660787 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:22:42.660787 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:42.660787 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:42.660787 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:42.660787 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:22:42.660787 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:42.660787 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:22:42.660787 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:22:42.660787 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:42.660787 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:22:42.661355 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:42.661355 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:22:42.661355 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:22:42.661355 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:42.661355 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:22:42.661355 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:42.661355 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:22:42.661669 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:22:42.661895 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_ha_failover {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:22:42.662076 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:22:42.662392 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:22:42.662977 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:22:42.663410 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:42.664192 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:22:42.664192 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:42.664192 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:22:42.664192 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:22:42.664192 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:42.664192 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:22:42.664192 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:42.664192 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:22:42.665053 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:42.665053 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:22:42.665053 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:22:42.665053 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:22:42.665053 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:42.665053 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:22:42.665053 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:42.665053 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:22:42.665053 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:42.665053 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:22:42.665053 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:42.665053 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:22:42.665053 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:42.665053 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:22:42.665053 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:22:42.665964 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:42.665964 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:22:42.665964 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:42.665964 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:22:42.665964 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:22:42.665964 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:42.665964 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:22:42.665964 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:42.665964 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:22:42.666365 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:42.666365 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:22:42.666365 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:42.666365 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:22:42.666365 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:22:42.666365 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:22:42.666365 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 15 10:22:42.666365 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:22:42.666365 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:22:42.666365 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:42.666365 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:42.666889 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:42.666889 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:22:42.666889 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:42.666889 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:42.666889 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:42.666889 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:22:42.666889 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:42.666889 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:22:42.666889 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:22:42.666889 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:42.666889 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:22:42.666889 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:42.666889 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:22:42.666889 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:22:42.667554 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:42.667554 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:22:42.667554 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:42.667554 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:22:42.667771 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:42.667771 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:22:42.667771 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:22:42.667771 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:22:42.667771 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:22:42.667771 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:22:42.667771 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:42.667771 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:22:42.667771 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:22:42.667771 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:42.667771 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:22:42.667771 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:22:42.667771 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:42.667771 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:42.667771 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:22:42.667771 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:42.667771 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:22:42.668527 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 15 10:22:42.668527 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:22:42.668527 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:22:42.668527 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:42.668527 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:42.668527 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:42.668527 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:22:42.668527 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:42.668527 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:42.668527 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:42.668527 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:22:42.668527 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:42.668527 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:22:42.668527 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:22:42.668527 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:42.668527 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:22:42.668527 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:42.668527 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:22:42.668527 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:22:42.669365 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:42.669365 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:22:42.669365 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:42.669365 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:22:42.670095 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:22:42.670459 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:22:42.670640 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:22:42.670824 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:22:42.671163 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:22:42.671457 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:22:42.671590 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:42.672207 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:22:42.672207 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:42.672207 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:22:42.672207 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:22:42.672207 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:42.672207 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:22:42.672207 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:42.672207 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:22:42.673461 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:42.673461 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:22:42.673461 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:42.673461 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:22:42.673461 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:22:42.673461 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:22:42.673461 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:22:42.673461 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:42.673461 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:42.673461 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:42.673461 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:22:42.673461 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:42.673864 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:42.673864 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:42.673864 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:22:42.673864 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:42.673864 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:22:42.673864 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:22:42.673864 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:42.673864 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:22:42.673864 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:42.673864 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:22:42.673864 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:22:42.673864 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:42.673864 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:22:42.674224 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:42.674224 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:22:42.674770 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:42.674770 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:22:42.674770 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:22:42.674770 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:22:42.674770 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:22:42.674770 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:22:42.674770 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:42.674770 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:22:42.674770 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:22:42.674770 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:42.674770 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:22:42.674770 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:22:42.674770 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:42.674770 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:42.674770 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:22:42.674770 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:42.675255 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:22:42.675255 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:22:42.675255 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:42.675255 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:42.675255 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:42.675255 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:22:42.675255 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:42.675255 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:42.675255 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:42.675255 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:22:42.675255 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:42.675255 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:22:42.675255 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:22:42.675255 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:42.675255 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:22:42.675255 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:42.675255 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:22:42.675255 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:22:42.675255 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:42.675827 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:22:42.675827 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:42.675827 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:22:47.536128 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:22:47.537160 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:22:47.537372 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:22:47.539161 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:22:47.579456 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:22:47.580084 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:47.580957 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:22:47.580957 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:47.580957 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:22:47.580957 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:22:47.580957 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:47.580957 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:22:47.580957 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:47.580957 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:22:47.582269 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_fdb_aging_settings failed after 0.044 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:47.582269 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:22:47.582269 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:47.582269 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:22:47.582269 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:22:47.582269 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:22:47.582269 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:22:47.582269 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:47.582269 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:47.582269 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:47.582269 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:22:47.582269 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:47.582864 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:47.582864 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:47.582864 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:22:47.582864 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:47.582864 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:22:47.582864 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:22:47.582864 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:47.582864 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:22:47.582864 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:47.582864 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:22:47.582864 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:22:47.582864 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:47.582864 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:22:47.583395 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:47.583395 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:22:47.583395 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:47.583395 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:22:47.583395 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:22:47.583395 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:22:47.583395 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:22:47.583395 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:22:47.583395 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:47.583395 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:22:47.583395 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:22:47.583395 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:47.583395 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:22:47.583395 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:22:47.583395 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:47.584111 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:47.584111 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:22:47.584111 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:47.584111 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:22:47.584111 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:22:47.584111 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:47.584111 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:47.584111 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:47.584111 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:22:47.584111 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:47.584111 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:47.584111 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:47.584111 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:22:47.584111 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:47.584111 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:22:47.584111 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:22:47.584111 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:47.584111 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:22:47.584111 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:47.585098 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:22:47.585098 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:22:47.585098 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:47.585098 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:22:47.585098 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:47.585098 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:22:47.644163 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:22:47.644609 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:22:47.645100 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:22:47.645393 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b027d63e-13a5-445b-84a0-8ae6223e05d5, routes=[], if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:22:47.645656 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:47.646407 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:22:47.646407 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:47.646407 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:22:47.646407 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:22:47.646407 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:47.646407 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:22:47.646407 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:47.646407 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:22:47.647711 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:47.647711 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:22:47.647711 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:47.647711 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:22:47.647711 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:22:47.647711 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:22:47.647711 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:22:47.647711 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:47.647711 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:47.647711 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:47.647711 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:22:47.647711 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:47.649098 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:47.649098 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:47.649098 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:22:47.649098 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:47.649098 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:22:47.649098 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:22:47.649098 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:47.649098 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:22:47.649098 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:47.649098 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:22:47.649098 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:22:47.649098 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:47.649098 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:22:47.649343 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:47.649343 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:22:47.649343 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:47.649343 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:22:47.649343 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:22:47.649343 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:22:47.649343 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:22:47.649343 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:22:47.649343 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:47.649343 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:22:47.649343 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:22:47.649343 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:47.649343 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:22:47.649343 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:22:47.649343 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:47.649623 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:47.649623 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:22:47.649623 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:47.649623 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:22:47.649623 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:22:47.649623 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:47.649623 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:47.649623 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:47.649623 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:22:47.649623 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:47.649623 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:47.649623 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:47.649623 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:22:47.649623 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:47.649623 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:22:47.649623 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:22:47.649623 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:47.649623 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:22:47.649623 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:47.649971 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:22:47.649971 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:22:47.649971 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:47.649971 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:22:47.649971 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:47.649971 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:22:47.657724 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:22:47.657724 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:22:47.657724 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:22:47.657724 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105458) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 15 10:22:47.657724 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:22:47.657724 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:47.658156 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:22:47.658156 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:47.658156 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:22:47.658156 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:22:47.658156 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:47.658156 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:22:47.658156 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:47.658156 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:22:47.659016 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:47.659016 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:22:47.659016 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:22:47.659016 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:22:47.659016 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:47.659016 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:22:47.659016 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:47.659016 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:22:47.659016 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:47.659016 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:22:47.659016 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:47.659016 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:22:47.659016 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:47.659016 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:22:47.659016 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:22:47.660272 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:47.660272 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:22:47.660272 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:47.660272 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:22:47.660272 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:22:47.660272 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:47.660272 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:22:47.660272 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:47.660272 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:22:47.660272 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:47.660272 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:22:47.660272 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:47.660272 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:22:47.660272 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:22:47.661027 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:22:47.661027 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:22:47.661027 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 15 10:22:47.661027 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:22:47.661027 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:22:47.661027 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:47.661027 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:47.661027 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:47.661027 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:22:47.661027 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:47.661027 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:47.661027 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:47.661027 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:22:47.661392 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:47.661392 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:22:47.661392 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:22:47.661392 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:47.661392 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:22:47.661392 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:47.661392 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:22:47.661392 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:22:47.661392 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:47.661392 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:22:47.661392 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:47.661392 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:22:47.662005 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:47.662005 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:22:47.662005 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:22:47.662005 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:22:47.662005 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:22:47.662005 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:22:47.662005 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:47.662005 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:22:47.662005 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:22:47.662005 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:47.662005 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:22:47.662005 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:22:47.662005 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:47.662005 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:47.662005 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:22:47.662005 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:47.662813 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:22:47.662813 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:22:47.662813 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^ May 15 10:22:47.662813 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:22:47.662813 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:22:47.662813 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:47.662813 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:47.662813 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:47.662813 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:22:47.662813 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:47.662813 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:47.662813 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:47.662813 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:22:47.662813 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:47.662813 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:22:47.662813 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:22:47.662813 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:47.662813 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:22:47.663864 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:47.663864 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:22:47.663864 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:22:47.663864 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:47.663864 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:22:47.663864 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:47.663864 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:22:47.663864 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:22:47.663864 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_ha_failover {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:22:47.663864 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:22:47.664181 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:22:47.664181 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:22:47.664181 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:47.664181 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:22:47.664181 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:47.664181 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:22:47.664181 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:22:47.664181 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:47.664181 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:22:47.664181 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:47.664181 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:22:47.664945 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:47.664945 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:22:47.664945 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:22:47.664945 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:22:47.664945 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:47.664945 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:22:47.664945 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:47.664945 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:22:47.664945 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:47.664945 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:22:47.664945 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:47.664945 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:22:47.664945 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:47.664945 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:22:47.664945 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:22:47.665336 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:47.665336 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:22:47.665336 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:47.665336 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:22:47.665336 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:22:47.665336 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:47.665336 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:22:47.665336 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:47.665336 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:22:47.666137 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:47.666137 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:22:47.666137 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:47.666137 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:22:47.666137 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:22:47.666137 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:22:47.666137 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 15 10:22:47.666137 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:22:47.666137 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:22:47.666137 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:47.666137 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:47.666422 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:47.666422 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:22:47.666422 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:47.666422 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:47.666422 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:47.666422 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:22:47.666422 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:47.666422 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:22:47.666422 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:22:47.666422 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:47.666422 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:22:47.666422 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:47.666422 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:22:47.666422 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:22:47.666777 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:47.666777 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:22:47.666777 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:47.666777 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:22:47.667669 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:47.667669 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:22:47.667669 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:22:47.667669 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:22:47.667669 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:22:47.667669 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:22:47.667669 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:47.667669 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:22:47.667669 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:22:47.667669 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:47.667669 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:22:47.667669 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:22:47.667669 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:47.667669 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:47.667669 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:22:47.667669 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:47.667669 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:22:47.668336 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 15 10:22:47.668336 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:22:47.668336 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:22:47.668336 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:47.668336 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:47.668336 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:47.668336 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:22:47.668336 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:47.668336 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:47.668336 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:47.668336 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:22:47.668336 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:47.668336 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:22:47.668336 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:22:47.668336 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:47.668336 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:22:47.668336 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:47.668336 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:22:47.668336 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:22:47.668855 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:47.668855 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:22:47.668855 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:47.668855 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:22:47.670723 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:22:47.671087 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:22:47.671279 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:22:47.671420 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:22:47.672061 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:22:47.672513 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:22:47.672758 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:47.673545 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:22:47.673545 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:47.673545 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:22:47.673545 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:22:47.673545 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:47.673545 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:22:47.673545 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:47.673545 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:22:47.674446 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:47.674446 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:22:47.674446 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:47.674446 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:22:47.674446 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:22:47.674446 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:22:47.674446 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:22:47.674446 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:47.674446 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:47.674446 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:47.674446 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:22:47.674446 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:47.675015 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:47.675015 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:47.675015 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:22:47.675015 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:47.675015 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:22:47.675015 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:22:47.675015 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:47.675015 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:22:47.675015 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:47.675015 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:22:47.675015 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:22:47.675015 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:47.675015 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:22:47.675627 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:47.675627 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:22:47.675627 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:47.675627 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:22:47.675627 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:22:47.675627 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:22:47.675627 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:22:47.675627 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:22:47.675627 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:47.675627 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:22:47.675627 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:22:47.675627 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:47.675627 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:22:47.675627 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:22:47.675627 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:47.676483 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:47.676483 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:22:47.676483 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:47.676483 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:22:47.676483 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:22:47.676483 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:47.676483 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:47.676483 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:47.676483 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:22:47.676483 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:47.676483 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:47.676483 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:47.676483 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:22:47.676483 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:47.676483 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:22:47.676483 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:22:47.676483 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:47.676483 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:22:47.676483 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:47.677230 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:22:47.677230 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:22:47.677230 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:47.677230 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:22:47.677230 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:47.677230 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:22:52.536997 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:22:52.537897 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:22:52.537897 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:22:52.538978 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:22:52.542761 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG dbcounter [-] [105458] Writing DB stats neutron:SELECT=240 {{(pid=105458) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} May 15 10:22:52.587005 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:22:52.587261 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:52.587947 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:22:52.587947 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:52.587947 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:22:52.587947 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:22:52.587947 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:52.587947 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:22:52.587947 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:52.587947 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:22:52.588974 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_fdb_aging_settings failed after 0.050 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:52.588974 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:22:52.588974 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:52.588974 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:22:52.588974 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:22:52.588974 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:22:52.588974 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:22:52.588974 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:52.588974 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:52.588974 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:52.588974 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:22:52.588974 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:52.589604 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:52.589604 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:52.589604 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:22:52.589604 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:52.589604 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:22:52.589604 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:22:52.589604 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:52.589604 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:22:52.589604 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:52.589604 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:22:52.589604 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:22:52.589604 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:52.589604 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:22:52.590149 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:52.590149 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:22:52.590658 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:52.590658 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:22:52.590658 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:22:52.590658 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:22:52.590658 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:22:52.590658 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:22:52.590658 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:52.590658 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:22:52.590658 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:22:52.590658 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:52.590658 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:22:52.590658 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:22:52.590658 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:52.590658 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:52.590658 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:22:52.590658 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:52.592382 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:22:52.592382 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:22:52.592382 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:52.592382 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:52.592382 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:52.592382 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:22:52.592382 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:52.592382 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:52.592382 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:52.592382 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:22:52.592382 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:52.592382 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:22:52.592382 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:22:52.592382 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:52.592382 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:22:52.592382 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:52.592382 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:22:52.592382 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:22:52.592382 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:52.593631 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:22:52.593631 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:52.593631 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:22:52.644879 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:22:52.645419 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:22:52.645619 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:22:52.646526 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b027d63e-13a5-445b-84a0-8ae6223e05d5, routes=[], if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:22:52.646882 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:52.647929 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:22:52.647929 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:52.647929 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:22:52.647929 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:22:52.647929 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:52.647929 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:22:52.647929 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:52.647929 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:22:52.649920 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:52.649920 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:22:52.649920 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:52.649920 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:22:52.649920 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:22:52.649920 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:22:52.649920 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:22:52.649920 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:52.649920 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:52.649920 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:52.649920 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:22:52.649920 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:52.650323 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:52.650323 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:52.650323 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:22:52.650323 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:52.650323 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:22:52.650323 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:22:52.650323 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:52.650323 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:22:52.650323 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:52.650323 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:22:52.650323 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:22:52.650323 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:52.650323 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:22:52.650771 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:52.650771 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:22:52.652221 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:52.652221 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:22:52.652221 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:22:52.652221 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:22:52.652221 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:22:52.652221 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:22:52.652221 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:52.652221 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:22:52.652221 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:22:52.652221 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:52.652221 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:22:52.652221 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:22:52.652221 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:52.652221 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:52.652221 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:22:52.652221 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:52.652793 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:22:52.652793 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:22:52.652793 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:52.652793 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:52.652793 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:52.652793 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:22:52.652793 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:52.652793 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:52.652793 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:52.652793 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:22:52.652793 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:52.652793 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:22:52.652793 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:22:52.652793 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:52.652793 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:22:52.652793 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:52.652793 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:22:52.652793 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:22:52.652793 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:52.653354 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:22:52.653354 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:52.653354 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:22:52.656290 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:22:52.656514 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:22:52.656656 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:22:52.656832 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105458) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 15 10:22:52.657221 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:22:52.657464 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:52.658015 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:22:52.658015 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:52.658015 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:22:52.658015 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:22:52.658015 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:52.658015 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:22:52.658015 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:52.658015 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:22:52.658826 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:52.658826 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:22:52.658826 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:22:52.658826 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:22:52.658826 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:52.658826 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:22:52.658826 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:52.658826 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:22:52.658826 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:52.658826 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:22:52.658826 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:52.658826 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:22:52.658826 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:52.658826 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:22:52.658826 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:22:52.659545 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:52.659545 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:22:52.659545 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:52.659545 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:22:52.659545 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:22:52.659545 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:52.659545 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:22:52.659545 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:52.659545 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:22:52.659783 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:52.659783 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:22:52.659783 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:52.659783 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:22:52.659783 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:22:52.659783 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:22:52.659783 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:22:52.659783 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 15 10:22:52.659783 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:22:52.659783 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:22:52.659783 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:52.659783 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:52.660109 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:52.660109 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:22:52.660109 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:52.660109 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:52.660109 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:52.660109 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:22:52.660109 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:52.660109 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:22:52.660109 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:22:52.660109 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:52.660109 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:22:52.660109 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:52.660109 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:22:52.660109 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:22:52.660434 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:52.660434 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:22:52.660434 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:52.660434 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:22:52.660964 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:52.660964 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:22:52.660964 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:22:52.660964 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:22:52.660964 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:22:52.660964 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:22:52.660964 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:52.660964 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:22:52.660964 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:22:52.660964 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:52.660964 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:22:52.660964 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:22:52.660964 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:52.660964 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:52.660964 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:22:52.660964 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:52.661410 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:22:52.661410 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:22:52.661410 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^ May 15 10:22:52.661410 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:22:52.661410 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:22:52.661410 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:52.661410 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:52.661410 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:52.661410 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:22:52.661410 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:52.661410 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:52.661410 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:52.661410 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:22:52.661410 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:52.661410 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:22:52.661410 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:22:52.661410 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:52.661410 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:22:52.661860 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:52.661860 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:22:52.661860 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:22:52.661860 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:52.661860 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:22:52.661860 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:52.661860 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:22:52.663174 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:22:52.663412 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_ha_failover {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:22:52.663532 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:22:52.663668 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:22:52.664024 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:22:52.664270 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:52.664792 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:22:52.664792 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:52.664792 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:22:52.664792 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:22:52.664792 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:52.664792 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:22:52.664792 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:52.664792 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:22:52.665559 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:52.665559 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:22:52.665559 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:22:52.665559 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:22:52.665559 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:52.665559 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:22:52.665559 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:52.665559 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:22:52.665559 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:52.665559 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:22:52.665559 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:52.665559 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:22:52.665559 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:52.665559 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:22:52.665559 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:22:52.665984 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:52.665984 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:22:52.665984 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:52.665984 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:22:52.665984 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:22:52.665984 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:52.665984 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:22:52.665984 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:52.665984 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:22:52.666426 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:52.666426 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:22:52.666426 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:52.666426 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:22:52.666426 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:22:52.666426 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:22:52.666426 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 15 10:22:52.666426 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:22:52.666426 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:22:52.666426 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:52.666426 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:52.666742 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:52.666742 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:22:52.666742 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:52.666742 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:52.666742 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:52.666742 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:22:52.666742 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:52.666742 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:22:52.666742 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:22:52.666742 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:52.666742 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:22:52.666742 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:52.666742 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:22:52.666742 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:22:52.667096 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:52.667096 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:22:52.667096 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:52.667096 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:22:52.667494 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:52.667494 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:22:52.667494 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:22:52.667494 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:22:52.667494 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:22:52.667494 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:22:52.667494 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:52.667494 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:22:52.667494 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:22:52.667494 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:52.667494 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:22:52.667494 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:22:52.667494 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:52.667494 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:52.667494 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:22:52.667494 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:52.667494 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:22:52.667941 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 15 10:22:52.667941 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:22:52.667941 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:22:52.667941 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:52.667941 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:52.667941 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:52.667941 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:22:52.667941 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:52.667941 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:52.667941 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:52.667941 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:22:52.667941 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:52.667941 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:22:52.667941 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:22:52.667941 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:52.667941 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:22:52.667941 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:52.667941 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:22:52.667941 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:22:52.668400 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:52.668400 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:22:52.668400 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:52.668400 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:22:52.671089 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:22:52.671311 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:22:52.671414 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:22:52.671587 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:22:52.671876 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:22:52.672097 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:22:52.672207 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:52.672695 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:22:52.672695 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:52.672695 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:22:52.672695 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:22:52.672695 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:52.672695 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:22:52.672695 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:52.672695 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:22:52.673774 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:52.673774 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:22:52.673774 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:52.673774 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:22:52.673774 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:22:52.673774 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:22:52.673774 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:22:52.673774 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:52.673774 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:52.673774 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:52.673774 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:22:52.673774 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:52.674117 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:52.674117 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:52.674117 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:22:52.674117 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:52.674117 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:22:52.674117 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:22:52.674117 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:52.674117 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:22:52.674117 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:52.674117 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:22:52.674117 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:22:52.674117 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:52.674117 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:22:52.674419 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:52.674419 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:22:52.674763 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:52.674763 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:22:52.674763 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:22:52.674763 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:22:52.674763 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:22:52.674763 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:22:52.674763 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:52.674763 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:22:52.674763 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:22:52.674763 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:52.674763 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:22:52.674763 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:22:52.674763 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:52.674763 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:52.674763 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:22:52.674763 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:52.676025 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:22:52.676025 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:22:52.676025 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:52.676025 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:52.676025 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:52.676025 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:22:52.676025 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:52.676025 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:52.676025 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:52.676025 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:22:52.676025 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:52.676025 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:22:52.676025 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:22:52.676025 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:52.676025 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:22:52.676025 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:52.676025 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:22:52.676025 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:22:52.676025 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:52.676479 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:22:52.676479 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:52.676479 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:22:57.537528 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:22:57.538493 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:22:57.538493 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:22:57.539116 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:22:57.574026 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:22:57.574485 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:57.575545 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:22:57.575545 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:57.575545 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:22:57.575545 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:22:57.575545 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:57.575545 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:22:57.575545 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:57.575545 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:22:57.577053 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_fdb_aging_settings failed after 0.037 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:57.577053 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:22:57.577053 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:57.577053 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:22:57.577053 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:22:57.577053 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:22:57.577053 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:22:57.577053 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:57.577053 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:57.577053 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:57.577053 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:22:57.577053 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:57.577595 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:57.577595 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:57.577595 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:22:57.577595 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:57.577595 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:22:57.577595 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:22:57.577595 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:57.577595 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:22:57.577595 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:57.577595 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:22:57.577595 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:22:57.577595 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:57.577595 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:22:57.578022 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:57.578022 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:22:57.579372 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:57.579372 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:22:57.579372 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:22:57.579372 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:22:57.579372 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:22:57.579372 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:22:57.579372 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:57.579372 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:22:57.579372 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:22:57.579372 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:57.579372 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:22:57.579372 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:22:57.579372 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:57.579372 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:57.579372 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:22:57.579372 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:57.579957 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:22:57.579957 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:22:57.579957 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:57.579957 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:57.579957 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:57.579957 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:22:57.579957 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:57.579957 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:57.579957 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:57.579957 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:22:57.579957 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:57.579957 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:22:57.579957 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:22:57.579957 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:57.579957 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:22:57.579957 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:57.579957 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:22:57.579957 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:22:57.579957 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:57.580511 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:22:57.580511 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:57.580511 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:22:57.645628 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:22:57.646457 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:22:57.646659 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:22:57.647555 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b027d63e-13a5-445b-84a0-8ae6223e05d5, routes=[], if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:22:57.647904 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:57.648653 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:22:57.648653 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:57.648653 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:22:57.648653 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:22:57.648653 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:57.648653 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:22:57.648653 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:57.648653 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:22:57.649820 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:57.649820 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:22:57.649820 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:57.649820 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:22:57.649820 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:22:57.649820 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:22:57.649820 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:22:57.649820 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:57.649820 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:57.649820 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:57.649820 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:22:57.649820 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:57.650397 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:57.650397 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:57.650397 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:22:57.650397 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:57.650397 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:22:57.650397 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:22:57.650397 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:57.650397 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:22:57.650397 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:57.650397 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:22:57.650397 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:22:57.650397 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:57.650397 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:22:57.650963 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:57.650963 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:22:57.651063 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:57.651063 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:22:57.651063 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:22:57.651063 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:22:57.651063 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:22:57.651063 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:22:57.651063 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:57.651063 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:22:57.651063 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:22:57.651063 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:57.651063 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:22:57.651063 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:22:57.651063 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:57.651063 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:57.651063 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:22:57.651063 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:57.651765 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:22:57.651765 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:22:57.651765 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:57.651765 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:57.651765 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:57.651765 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:22:57.651765 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:57.651765 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:57.651765 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:57.651765 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:22:57.651765 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:57.651765 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:22:57.651765 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:22:57.651765 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:57.651765 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:22:57.651765 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:57.651765 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:22:57.651765 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:22:57.651765 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:57.652658 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:22:57.652658 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:57.652658 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:22:57.656804 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:22:57.657491 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:22:57.657608 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:22:57.657800 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105458) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 15 10:22:57.658520 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:22:57.658974 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:57.660377 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:22:57.660377 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:57.660377 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:22:57.660377 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:22:57.660377 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:57.660377 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:22:57.660377 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:57.660377 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:22:57.661250 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:57.661250 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:22:57.661250 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:22:57.661250 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:22:57.661250 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:57.661250 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:22:57.661250 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:57.661250 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:22:57.661250 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:57.661250 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:22:57.661250 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:57.661250 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:22:57.661250 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:57.661250 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:22:57.661250 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:22:57.661937 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:57.661937 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:22:57.661937 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:57.661937 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:22:57.661937 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:22:57.661937 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:57.661937 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:22:57.661937 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:57.661937 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:22:57.662632 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:57.662632 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:22:57.662632 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:57.662632 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:22:57.662632 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:22:57.662632 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:22:57.662632 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:22:57.662632 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 15 10:22:57.662632 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:22:57.662632 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:22:57.662632 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:57.662632 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:57.663036 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:57.663036 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:22:57.663036 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:57.663036 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:57.663036 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:57.663036 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:22:57.663036 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:57.663036 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:22:57.663036 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:22:57.663036 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:57.663036 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:22:57.663036 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:57.663036 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:22:57.663036 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:22:57.664046 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:57.664046 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:22:57.664046 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:57.664046 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:22:57.664046 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:57.664046 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:22:57.664046 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:22:57.664046 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:22:57.664046 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:22:57.664046 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:22:57.664046 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:57.664046 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:22:57.664046 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:22:57.664046 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:57.664487 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:22:57.664487 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:22:57.664487 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:57.664487 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:57.664487 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:22:57.664487 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:57.664487 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:22:57.664487 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:22:57.664487 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^ May 15 10:22:57.664487 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:22:57.664487 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:22:57.664487 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:57.664487 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:57.664487 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:57.664487 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:22:57.664487 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:57.664487 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:57.664487 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:57.664487 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:22:57.665023 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:57.665023 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:22:57.665023 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:22:57.665023 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:57.665023 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:22:57.665023 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:57.665023 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:22:57.665023 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:22:57.665023 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:57.665023 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:22:57.665023 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:57.665023 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:22:57.665023 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:22:57.665023 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_ha_failover {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:22:57.665436 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:22:57.665436 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:22:57.665500 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:22:57.665942 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:57.666419 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:22:57.666419 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:57.666419 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:22:57.666419 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:22:57.666419 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:57.666419 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:22:57.666419 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:57.666419 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:22:57.667341 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:57.667341 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:22:57.667341 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:22:57.667341 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:22:57.667341 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:57.667341 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:22:57.667341 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:57.667341 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:22:57.667341 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:57.667341 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:22:57.667341 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:57.667341 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:22:57.667341 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:57.667341 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:22:57.667341 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:22:57.668405 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:57.668405 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:22:57.668405 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:57.668405 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:22:57.668405 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:22:57.668405 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:57.668405 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:22:57.668405 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:57.668405 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:22:57.668648 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:57.668648 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:22:57.668648 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:57.668648 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:22:57.668648 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:22:57.668648 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:22:57.668648 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 15 10:22:57.668648 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:22:57.668648 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:22:57.668648 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:57.668648 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:57.669012 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:57.669012 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:22:57.669012 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:57.669012 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:57.669012 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:57.669012 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:22:57.669012 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:57.669012 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:22:57.669012 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:22:57.669012 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:57.669012 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:22:57.669012 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:57.669012 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:22:57.669012 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:22:57.669388 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:57.669388 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:22:57.669388 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:57.669388 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:22:57.669711 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:57.669711 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:22:57.669711 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:22:57.669711 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:22:57.669711 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:22:57.669711 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:22:57.669711 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:57.669711 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:22:57.669711 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:22:57.669711 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:57.669711 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:22:57.669711 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:22:57.669711 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:57.669711 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:57.669711 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:22:57.669711 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:57.669711 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:22:57.670290 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 15 10:22:57.670290 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:22:57.670290 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:22:57.670290 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:57.670290 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:57.670290 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:57.670290 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:22:57.670290 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:57.670290 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:57.670290 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:57.670290 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:22:57.670290 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:57.670290 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:22:57.670290 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:22:57.670290 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:57.670290 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:22:57.670290 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:57.670290 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:22:57.670290 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:22:57.671266 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:57.671266 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:22:57.671266 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:57.671266 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:22:57.671504 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:22:57.671952 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:22:57.672069 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:22:57.672275 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:22:57.672767 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:22:57.673041 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:22:57.673176 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:57.673886 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:22:57.673886 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:57.673886 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:22:57.673886 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:22:57.673886 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:57.673886 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:22:57.673886 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:57.673886 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:22:57.674901 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:57.674901 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:22:57.674901 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:57.674901 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:22:57.674901 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:22:57.674901 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:22:57.674901 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:22:57.674901 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:57.674901 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:57.674901 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:57.674901 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:22:57.674901 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:57.675299 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:22:57.675299 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:57.675299 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:22:57.675299 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:57.675299 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:22:57.675299 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:22:57.675299 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:57.675299 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:22:57.675299 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:57.675299 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:22:57.675299 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:22:57.675299 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:57.675299 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:22:57.675659 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:57.675659 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:22:57.676242 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:57.676242 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:22:57.676242 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:22:57.676242 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:22:57.676242 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:22:57.676242 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:22:57.676242 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:57.676242 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:22:57.676242 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:22:57.676242 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:57.676242 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:22:57.676242 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:22:57.676242 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:22:57.676242 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:22:57.676242 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:22:57.676242 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:22:57.676762 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:22:57.676762 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:22:57.676762 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:57.676762 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:57.676762 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:22:57.676762 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:22:57.676762 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:22:57.676762 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:22:57.676762 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:22:57.676762 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:22:57.676762 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:22:57.676762 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:22:57.676762 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:22:57.676762 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:22:57.676762 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:22:57.676762 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:22:57.676762 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:22:57.676762 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:22:57.676762 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:22:57.677441 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:22:57.677441 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:22:57.677441 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:23:02.538196 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:23:02.538892 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:23:02.538942 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:23:02.540048 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:23:02.576268 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:23:02.576533 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:02.577218 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:23:02.577218 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:23:02.577218 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:23:02.577218 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:23:02.577218 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:23:02.577218 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:23:02.577218 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:02.577218 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:23:02.577982 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_fdb_aging_settings failed after 0.038 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:02.577982 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:23:02.577982 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:23:02.577982 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:23:02.577982 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:23:02.577982 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:23:02.577982 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:23:02.577982 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:23:02.577982 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:23:02.577982 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:23:02.577982 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:23:02.577982 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:23:02.578744 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:23:02.578744 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:23:02.578744 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:23:02.578744 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:23:02.578744 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:23:02.578744 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:23:02.578744 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:23:02.578744 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:23:02.578744 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:23:02.578744 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:23:02.578744 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:23:02.578744 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:23:02.578744 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:23:02.579511 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:02.579511 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:23:02.579511 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:02.579511 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:23:02.579511 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:23:02.579511 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:23:02.579511 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:23:02.579511 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:23:02.579511 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:23:02.579511 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:23:02.579511 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:23:02.579511 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:23:02.579511 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:23:02.579511 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:23:02.579511 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:23:02.580367 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:23:02.580367 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:23:02.580367 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:23:02.580367 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:23:02.580367 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:23:02.580367 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:23:02.580367 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:23:02.580367 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:23:02.580367 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:23:02.580367 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:23:02.580367 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:23:02.580367 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:23:02.580367 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:23:02.580367 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:23:02.580367 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:23:02.580367 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:23:02.580367 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:23:02.580367 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:23:02.580367 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:23:02.580863 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:23:02.580863 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:23:02.580863 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:23:02.580863 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:23:02.580863 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:02.580863 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:23:02.646745 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:23:02.647215 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:23:02.647301 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:23:02.648017 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b027d63e-13a5-445b-84a0-8ae6223e05d5, routes=[], if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:23:02.648307 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:02.649071 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:23:02.649071 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:23:02.649071 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:23:02.649071 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:23:02.649071 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:23:02.649071 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:23:02.649071 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:02.649071 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:23:02.649933 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:02.649933 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:23:02.649933 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:23:02.649933 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:23:02.649933 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:23:02.649933 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:23:02.649933 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:23:02.649933 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:23:02.649933 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:23:02.649933 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:23:02.649933 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:23:02.649933 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:23:02.650478 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:23:02.650478 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:23:02.650478 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:23:02.650478 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:23:02.650478 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:23:02.650478 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:23:02.650478 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:23:02.650478 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:23:02.650478 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:23:02.650478 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:23:02.650478 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:23:02.650478 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:23:02.650478 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:23:02.650998 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:02.650998 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:23:02.650998 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:02.650998 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:23:02.650998 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:23:02.650998 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:23:02.650998 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:23:02.650998 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:23:02.650998 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:23:02.650998 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:23:02.650998 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:23:02.650998 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:23:02.650998 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:23:02.650998 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:23:02.650998 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:23:02.651653 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:23:02.651653 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:23:02.651653 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:23:02.651653 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:23:02.651653 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:23:02.651653 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:23:02.651653 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:23:02.651653 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:23:02.651653 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:23:02.651653 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:23:02.651653 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:23:02.651653 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:23:02.651653 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:23:02.651653 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:23:02.651653 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:23:02.651653 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:23:02.651653 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:23:02.651653 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:23:02.651653 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:23:02.652458 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:23:02.652458 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:23:02.652458 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:23:02.652458 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:23:02.652458 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:02.652458 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:23:02.657384 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:23:02.657780 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:23:02.657925 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:23:02.658050 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105458) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 15 10:23:02.658412 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:23:02.658653 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:02.659637 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:23:02.659637 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:23:02.659637 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:23:02.659637 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:23:02.659637 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:23:02.659637 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:23:02.659637 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:02.659637 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:23:02.661918 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:02.661918 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:23:02.661918 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:23:02.661918 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:23:02.661918 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:23:02.661918 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:23:02.661918 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:23:02.661918 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:23:02.661918 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:23:02.661918 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:23:02.661918 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:23:02.661918 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:23:02.661918 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:23:02.661918 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:23:02.661918 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:23:02.662938 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:23:02.662938 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:23:02.662938 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:23:02.662938 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:23:02.662938 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:23:02.662938 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:23:02.662938 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:23:02.662938 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:02.662938 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:23:02.662938 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:02.662938 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:23:02.662938 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:23:02.662938 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:23:02.662938 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:23:02.663822 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:23:02.663822 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:23:02.663822 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 15 10:23:02.663822 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:23:02.663822 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:23:02.663822 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:23:02.663822 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:23:02.663822 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:23:02.663822 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:23:02.663822 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:23:02.663822 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:23:02.663822 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:23:02.663822 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:23:02.664533 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:23:02.664533 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:23:02.664533 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:23:02.664533 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:23:02.664533 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:23:02.664533 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:23:02.664533 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:23:02.664533 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:23:02.664533 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:23:02.664533 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:23:02.664533 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:02.664533 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:23:02.666716 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:02.666716 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:23:02.666716 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:23:02.666716 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:23:02.666716 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:23:02.666716 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:23:02.666716 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:23:02.666716 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:23:02.666716 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:23:02.666716 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:23:02.666716 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:23:02.666716 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:23:02.666716 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:23:02.666716 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:23:02.666716 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:23:02.666716 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:23:02.668277 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:23:02.668277 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:23:02.668277 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^ May 15 10:23:02.668277 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:23:02.668277 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:23:02.668277 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:23:02.668277 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:23:02.668277 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:23:02.668277 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:23:02.668277 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:23:02.668277 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:23:02.668277 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:23:02.668277 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:23:02.668277 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:23:02.668277 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:23:02.668277 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:23:02.668277 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:23:02.668277 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:23:02.669422 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:23:02.669422 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:23:02.669422 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:23:02.669422 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:23:02.669422 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:23:02.669422 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:02.669422 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:23:02.669422 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:23:02.669422 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_ha_failover {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:23:02.669422 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:23:02.670238 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:23:02.670238 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:23:02.670238 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:02.670238 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:23:02.670238 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:23:02.670238 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:23:02.670238 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:23:02.670238 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:23:02.670238 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:23:02.670238 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:02.670238 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:23:02.670238 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:02.670983 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:23:02.670983 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:23:02.670983 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:23:02.670983 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:23:02.670983 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:23:02.670983 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:23:02.670983 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:23:02.670983 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:23:02.670983 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:23:02.670983 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:23:02.670983 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:23:02.670983 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:23:02.670983 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:23:02.670983 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:23:02.670983 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:23:02.670983 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:23:02.670983 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:23:02.671513 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:23:02.671513 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:23:02.671513 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:23:02.671513 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:23:02.671513 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:02.671513 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:23:02.671513 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:02.671513 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:23:02.671513 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:23:02.671513 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:23:02.671513 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:23:02.671513 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:23:02.671513 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 15 10:23:02.671872 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:23:02.671872 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:23:02.671872 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:23:02.671872 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:23:02.671872 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:23:02.671872 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:23:02.671872 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:23:02.671872 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:23:02.671872 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:23:02.671872 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:23:02.671872 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:23:02.671872 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:23:02.671872 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:23:02.672095 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:23:02.672095 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:23:02.672095 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:23:02.672095 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:23:02.672095 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:23:02.672095 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:23:02.672095 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:23:02.672095 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:02.672095 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:23:02.672095 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:02.672095 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:23:02.672095 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:23:02.672323 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:23:02.672323 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:23:02.672323 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:23:02.672323 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:23:02.672323 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:23:02.672323 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:23:02.672323 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:23:02.672323 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:23:02.672323 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:23:02.672323 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:23:02.672323 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:23:02.672323 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:23:02.672323 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:23:02.672323 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:23:02.672323 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 15 10:23:02.672323 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:23:02.672323 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:23:02.672323 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:23:02.672323 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:23:02.672645 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:23:02.672645 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:23:02.672645 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:23:02.672645 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:23:02.672645 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:23:02.672645 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:23:02.672645 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:23:02.672645 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:23:02.672645 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:23:02.672645 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:23:02.672645 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:23:02.672645 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:23:02.672645 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:23:02.672645 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:23:02.672645 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:23:02.672645 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:23:02.672645 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:02.672645 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:23:02.673052 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:23:02.673052 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:23:02.673052 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:23:02.673052 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:23:02.673052 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:23:02.673052 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:23:02.673184 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:02.673831 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:23:02.673831 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:23:02.673831 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:23:02.673831 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:23:02.673831 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:23:02.673831 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:23:02.673831 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:02.673831 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:23:02.674613 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:02.674613 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:23:02.674613 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:23:02.674613 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:23:02.674613 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:23:02.674613 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:23:02.674613 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:23:02.674613 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:23:02.674613 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:23:02.674613 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:23:02.674613 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:23:02.674613 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:23:02.675194 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:23:02.675194 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:23:02.675194 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:23:02.675194 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:23:02.675194 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:23:02.675194 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:23:02.675194 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:23:02.675194 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:23:02.675194 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:23:02.675194 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:23:02.675194 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:23:02.675194 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:23:02.675194 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:23:02.675739 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:02.675739 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:23:02.675739 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:02.675739 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:23:02.675739 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:23:02.675739 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:23:02.675739 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:23:02.675739 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:23:02.675739 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:23:02.675739 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:23:02.675739 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:23:02.675739 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:23:02.675739 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:23:02.675739 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:23:02.675739 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:23:02.676364 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:23:02.676364 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:23:02.676364 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:23:02.676364 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:23:02.676364 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:23:02.676364 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:23:02.676364 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:23:02.676364 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:23:02.676364 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:23:02.676364 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:23:02.676364 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:23:02.676364 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:23:02.676364 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:23:02.676364 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:23:02.676364 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:23:02.676364 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:23:02.676364 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:23:02.676364 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:23:02.676364 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:23:02.677100 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:23:02.677100 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:23:02.677100 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:23:02.677100 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:23:02.677100 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:02.677100 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:23:07.538883 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:23:07.539531 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:23:07.539883 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:23:07.540865 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:23:07.581507 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:23:07.581829 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:07.582562 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:23:07.582562 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:23:07.582562 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:23:07.582562 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:23:07.582562 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:23:07.582562 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:23:07.582562 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:07.582562 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:23:07.583776 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_fdb_aging_settings failed after 0.043 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:07.583776 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:23:07.583776 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:23:07.583776 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:23:07.583776 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:23:07.583776 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:23:07.583776 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:23:07.583776 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:23:07.583776 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:23:07.583776 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:23:07.583776 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:23:07.583776 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:23:07.584134 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:23:07.584134 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:23:07.584134 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:23:07.584134 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:23:07.584134 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:23:07.584134 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:23:07.584134 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:23:07.584134 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:23:07.584134 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:23:07.584134 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:23:07.584134 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:23:07.584134 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:23:07.584134 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:23:07.584391 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:07.584391 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:23:07.584956 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:07.584956 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:23:07.584956 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:23:07.584956 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:23:07.584956 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:23:07.584956 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:23:07.584956 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:23:07.584956 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:23:07.584956 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:23:07.584956 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:23:07.584956 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:23:07.584956 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:23:07.584956 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:23:07.584956 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:23:07.584956 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:23:07.584956 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:23:07.585347 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:23:07.585347 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:23:07.585347 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:23:07.585347 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:23:07.585347 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:23:07.585347 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:23:07.585347 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:23:07.585347 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:23:07.585347 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:23:07.585347 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:23:07.585347 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:23:07.585347 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:23:07.585347 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:23:07.585347 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:23:07.585347 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:23:07.585347 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:23:07.585347 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:23:07.585347 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:23:07.585347 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:23:07.585807 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:23:07.585807 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:07.585807 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:23:07.647532 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:23:07.648096 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:23:07.648390 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:23:07.649241 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b027d63e-13a5-445b-84a0-8ae6223e05d5, routes=[], if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:23:07.649547 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:07.650271 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:23:07.650271 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:23:07.650271 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:23:07.650271 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:23:07.650271 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:23:07.650271 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:23:07.650271 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:07.650271 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:23:07.651038 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:07.651038 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:23:07.651038 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:23:07.651038 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:23:07.651038 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:23:07.651038 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:23:07.651038 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:23:07.651038 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:23:07.651038 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:23:07.651038 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:23:07.651038 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:23:07.651038 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:23:07.651380 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:23:07.651380 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:23:07.651380 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:23:07.651380 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:23:07.651380 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:23:07.651380 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:23:07.651380 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:23:07.651380 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:23:07.651380 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:23:07.651380 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:23:07.651380 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:23:07.651380 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:23:07.651380 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:23:07.651773 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:07.651773 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:23:07.651919 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:07.651919 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:23:07.651919 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:23:07.651919 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:23:07.651919 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:23:07.651919 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:23:07.651919 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:23:07.651919 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:23:07.651919 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:23:07.651919 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:23:07.651919 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:23:07.651919 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:23:07.651919 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:23:07.651919 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:23:07.651919 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:23:07.651919 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:23:07.652400 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:23:07.652400 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:23:07.652400 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:23:07.652400 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:23:07.652400 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:23:07.652400 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:23:07.652400 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:23:07.652400 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:23:07.652400 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:23:07.652400 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:23:07.652400 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:23:07.652400 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:23:07.652400 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:23:07.652400 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:23:07.652400 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:23:07.652400 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:23:07.652400 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:23:07.652400 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:23:07.652400 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:23:07.653062 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:23:07.653062 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:07.653062 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:23:07.657778 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:23:07.658054 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:23:07.658183 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:23:07.658357 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105458) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 15 10:23:07.658780 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:23:07.659024 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:07.659583 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:23:07.659583 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:23:07.659583 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:23:07.659583 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:23:07.659583 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:23:07.659583 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:23:07.659583 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:07.659583 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:23:07.660619 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:07.660619 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:23:07.660619 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:23:07.660619 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:23:07.660619 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:23:07.660619 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:23:07.660619 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:23:07.660619 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:23:07.660619 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:23:07.660619 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:23:07.660619 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:23:07.660619 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:23:07.660619 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:23:07.660619 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:23:07.660619 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:23:07.661203 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:23:07.661203 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:23:07.661203 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:23:07.661203 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:23:07.661203 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:23:07.661203 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:23:07.661203 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:23:07.661203 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:07.661203 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:23:07.661798 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:07.661798 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:23:07.661798 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:23:07.661798 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:23:07.661798 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:23:07.661798 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:23:07.661798 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:23:07.661798 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 15 10:23:07.661798 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:23:07.661798 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:23:07.661798 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:23:07.661798 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:23:07.662146 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:23:07.662146 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:23:07.662146 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:23:07.662146 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:23:07.662146 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:23:07.662146 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:23:07.662146 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:23:07.662146 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:23:07.662146 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:23:07.662146 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:23:07.662146 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:23:07.662146 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:23:07.662146 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:23:07.662146 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:23:07.662504 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:23:07.662504 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:23:07.662504 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:07.662504 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:23:07.663376 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:07.663376 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:23:07.663376 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:23:07.663376 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:23:07.663376 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:23:07.663376 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:23:07.663376 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:23:07.663376 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:23:07.663376 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:23:07.663376 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:23:07.663376 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:23:07.663376 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:23:07.663376 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:23:07.663376 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:23:07.663376 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:23:07.663376 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:23:07.663932 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:23:07.663932 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:23:07.663932 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^ May 15 10:23:07.663932 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:23:07.663932 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:23:07.663932 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:23:07.663932 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:23:07.663932 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:23:07.663932 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:23:07.663932 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:23:07.663932 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:23:07.663932 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:23:07.663932 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:23:07.663932 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:23:07.663932 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:23:07.663932 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:23:07.663932 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:23:07.663932 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:23:07.664423 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:23:07.664423 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:23:07.664423 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:23:07.664423 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:23:07.664423 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:23:07.664423 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:07.664423 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:23:07.664997 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:23:07.665279 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_ha_failover {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:23:07.665432 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:23:07.665612 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:23:07.666038 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:23:07.666331 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:07.666951 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:23:07.666951 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:23:07.666951 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:23:07.666951 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:23:07.666951 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:23:07.666951 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:23:07.666951 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:07.666951 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:23:07.667981 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:07.667981 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:23:07.667981 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:23:07.667981 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:23:07.667981 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:23:07.667981 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:23:07.667981 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:23:07.667981 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:23:07.667981 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:23:07.667981 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:23:07.667981 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:23:07.667981 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:23:07.667981 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:23:07.667981 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:23:07.667981 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:23:07.668587 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:23:07.668587 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:23:07.668587 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:23:07.668587 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:23:07.668587 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:23:07.668587 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:23:07.668587 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:23:07.668587 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:07.668587 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:23:07.669290 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:07.669290 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:23:07.669290 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:23:07.669290 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:23:07.669290 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:23:07.669290 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:23:07.669290 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 15 10:23:07.669290 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:23:07.669290 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:23:07.669290 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:23:07.669290 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:23:07.669625 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:23:07.669625 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:23:07.669625 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:23:07.669625 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:23:07.669625 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:23:07.669625 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:23:07.669625 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:23:07.669625 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:23:07.669625 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:23:07.669625 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:23:07.669625 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:23:07.669625 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:23:07.669625 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:23:07.669625 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:23:07.670028 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:23:07.670028 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:23:07.670028 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:07.670028 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:23:07.670946 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:07.670946 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:23:07.670946 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:23:07.670946 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:23:07.670946 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:23:07.670946 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:23:07.670946 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:23:07.670946 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:23:07.670946 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:23:07.670946 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:23:07.670946 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:23:07.670946 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:23:07.670946 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:23:07.670946 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:23:07.670946 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:23:07.670946 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:23:07.670946 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:23:07.671476 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 15 10:23:07.671476 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:23:07.671476 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:23:07.671476 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:23:07.671476 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:23:07.671476 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:23:07.671476 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:23:07.671476 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:23:07.671476 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:23:07.671476 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:23:07.671476 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:23:07.671476 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:23:07.671476 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:23:07.671476 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:23:07.671476 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:23:07.671476 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:23:07.671476 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:23:07.671476 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:23:07.671476 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:23:07.672102 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:23:07.672102 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:23:07.672102 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:07.672102 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:23:07.672311 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:23:07.672619 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:23:07.672789 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:23:07.673025 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:23:07.673399 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:23:07.673701 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:23:07.673867 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:07.674958 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:23:07.674958 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:23:07.674958 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:23:07.674958 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:23:07.674958 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:23:07.674958 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:23:07.674958 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:07.674958 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:23:07.676387 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:07.676387 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:23:07.676387 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:23:07.676387 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:23:07.676387 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:23:07.676387 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:23:07.676387 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:23:07.676387 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:23:07.676387 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:23:07.676387 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:23:07.676387 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:23:07.676387 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:23:07.676840 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:23:07.676840 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:23:07.676840 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:23:07.676840 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:23:07.676840 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:23:07.676840 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:23:07.676840 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:23:07.676840 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:23:07.676840 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:23:07.676840 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:23:07.676840 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:23:07.676840 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:23:07.676840 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:23:07.677225 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:07.677225 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:23:07.677293 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:07.677293 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:23:07.677293 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:23:07.677293 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:23:07.677293 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:23:07.677293 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:23:07.677293 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:23:07.677293 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:23:07.677293 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:23:07.677293 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:23:07.677293 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:23:07.677293 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:23:07.677293 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:23:07.677293 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:23:07.677293 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:23:07.677293 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:23:07.677931 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:23:07.677931 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:23:07.677931 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:23:07.677931 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:23:07.677931 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:23:07.677931 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:23:07.677931 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:23:07.677931 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:23:07.677931 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:23:07.677931 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:23:07.677931 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:23:07.677931 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:23:07.677931 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:23:07.677931 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:23:07.677931 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:23:07.677931 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:23:07.677931 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:23:07.677931 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:23:07.677931 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:23:07.678628 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:23:07.678628 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:07.678628 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:23:12.539612 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:23:12.540548 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:23:12.540764 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:23:12.542159 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:23:12.585313 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:23:12.585745 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:12.586728 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:23:12.586728 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:23:12.586728 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:23:12.586728 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:23:12.586728 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:23:12.586728 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:23:12.586728 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:12.586728 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:23:12.587547 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_fdb_aging_settings failed after 0.046 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:12.587547 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:23:12.587547 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:23:12.587547 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:23:12.587547 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:23:12.587547 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:23:12.587547 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:23:12.587547 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:23:12.587547 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:23:12.587547 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:23:12.587547 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:23:12.587547 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:23:12.587884 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:23:12.587884 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:23:12.587884 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:23:12.587884 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:23:12.587884 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:23:12.587884 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:23:12.587884 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:23:12.587884 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:23:12.587884 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:23:12.587884 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:23:12.587884 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:23:12.587884 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:23:12.587884 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:23:12.588484 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:12.588484 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:23:12.588484 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:12.588484 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:23:12.588484 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:23:12.588484 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:23:12.588484 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:23:12.588484 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:23:12.588484 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:23:12.588484 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:23:12.588484 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:23:12.588484 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:23:12.588484 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:23:12.588484 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:23:12.588484 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:23:12.588859 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:23:12.588859 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:23:12.588859 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:23:12.588859 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 15 10:23:12.588859 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:23:12.588859 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:23:12.588859 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:23:12.588859 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:23:12.588859 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:23:12.588859 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:23:12.588859 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:23:12.588859 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:23:12.588859 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:23:12.588859 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:23:12.588859 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:23:12.588859 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:23:12.588859 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:23:12.588859 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:23:12.588859 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:23:12.589283 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:23:12.589283 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:23:12.589283 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:23:12.589283 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:23:12.589283 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:12.589283 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:23:12.648178 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:23:12.648849 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:23:12.649077 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:23:12.650008 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-b027d63e-13a5-445b-84a0-8ae6223e05d5, routes=[], if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:23:12.650330 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:12.651415 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:23:12.651415 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:23:12.651415 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:23:12.651415 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:23:12.651415 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:23:12.651415 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:23:12.651415 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:12.651415 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:23:12.652702 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:12.652702 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:23:12.652702 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:23:12.652702 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:23:12.652702 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:23:12.652702 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:23:12.652702 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:23:12.652702 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:23:12.652702 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:23:12.652702 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:23:12.652702 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:23:12.652702 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:23:12.654552 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:23:12.654552 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:23:12.654552 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:23:12.654552 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:23:12.654552 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:23:12.654552 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:23:12.654552 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:23:12.654552 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:23:12.654552 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:23:12.654552 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:23:12.654552 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:23:12.654552 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:23:12.654552 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:23:12.655566 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:12.655566 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:23:12.655566 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:12.655566 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:23:12.655566 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:23:12.655566 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:23:12.655566 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:23:12.655566 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:23:12.655566 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:23:12.655566 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:23:12.655566 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:23:12.655566 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:23:12.655566 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:23:12.655566 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:23:12.655566 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:23:12.655965 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:23:12.655965 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:23:12.655965 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:23:12.655965 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 15 10:23:12.655965 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:23:12.655965 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:23:12.655965 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:23:12.655965 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:23:12.655965 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:23:12.655965 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:23:12.655965 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:23:12.655965 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:23:12.655965 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:23:12.655965 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:23:12.655965 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:23:12.655965 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:23:12.655965 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:23:12.655965 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:23:12.655965 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:23:12.656381 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:23:12.656381 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:23:12.656381 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:23:12.656381 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:23:12.656381 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:12.656381 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:23:12.658154 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:23:12.658445 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:23:12.658567 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:23:12.658742 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting fip-distributed flag in NB_Global to False {{(pid=105458) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 15 10:23:12.659174 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:23:12.659386 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:12.659893 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:23:12.659893 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:23:12.659893 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:23:12.659893 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:23:12.659893 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:23:12.659893 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:23:12.659893 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:12.659893 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:23:12.660694 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:12.660694 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:23:12.660694 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:23:12.660694 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:23:12.660694 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:23:12.660694 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:23:12.660694 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:23:12.660694 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:23:12.660694 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:23:12.660694 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:23:12.660694 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:23:12.660694 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:23:12.660694 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:23:12.660694 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:23:12.660694 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:23:12.661109 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:23:12.661109 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:23:12.661109 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:23:12.661109 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:23:12.661109 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:23:12.661109 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:23:12.661109 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:23:12.661109 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:12.661109 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:23:12.661646 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:12.661646 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:23:12.661646 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:23:12.661646 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:23:12.661646 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:23:12.661646 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:23:12.661646 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:23:12.661646 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 15 10:23:12.661646 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:23:12.661646 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:23:12.661646 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:23:12.661646 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:23:12.661981 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:23:12.661981 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:23:12.661981 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:23:12.661981 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:23:12.661981 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:23:12.661981 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:23:12.661981 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:23:12.661981 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:23:12.661981 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:23:12.661981 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:23:12.661981 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:23:12.661981 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:23:12.661981 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:23:12.661981 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:23:12.662297 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:23:12.662297 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:23:12.662297 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:12.662297 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:23:12.662962 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:12.662962 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:23:12.662962 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:23:12.662962 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:23:12.662962 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:23:12.662962 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:23:12.662962 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:23:12.662962 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:23:12.662962 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:23:12.662962 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:23:12.662962 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:23:12.662962 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:23:12.662962 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:23:12.662962 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:23:12.662962 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:23:12.662962 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:23:12.663464 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 15 10:23:12.663464 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 15 10:23:12.663464 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^ May 15 10:23:12.663464 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:23:12.663464 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:23:12.663464 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:23:12.663464 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:23:12.663464 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:23:12.663464 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:23:12.663464 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:23:12.663464 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:23:12.663464 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:23:12.663464 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:23:12.663464 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:23:12.663464 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:23:12.663464 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:23:12.663464 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:23:12.663464 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:23:12.663892 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:23:12.663892 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:23:12.663892 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:23:12.663892 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:23:12.663892 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:23:12.663892 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:12.663892 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:23:12.665306 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:23:12.665576 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_ha_failover {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:23:12.665691 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:23:12.665898 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:23:12.666182 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:23:12.666402 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:12.667053 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:23:12.667053 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:23:12.667053 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:23:12.667053 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:23:12.667053 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:23:12.667053 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:23:12.667053 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:12.667053 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:23:12.667828 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:12.667828 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 15 10:23:12.667828 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:23:12.667828 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:23:12.667828 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:23:12.667828 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:23:12.667828 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:23:12.667828 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 15 10:23:12.667828 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:23:12.667828 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 15 10:23:12.667828 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:23:12.667828 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 15 10:23:12.667828 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:23:12.667828 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 15 10:23:12.667828 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 15 10:23:12.668217 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:23:12.668217 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 15 10:23:12.668217 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:23:12.668217 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 15 10:23:12.668217 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 15 10:23:12.668217 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:23:12.668217 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 15 10:23:12.668217 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:12.668217 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.command May 15 10:23:12.668863 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:12.668863 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:23:12.668863 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:23:12.668863 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:23:12.668863 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:23:12.668863 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:23:12.668863 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 15 10:23:12.668863 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:23:12.668863 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:23:12.668863 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:23:12.668863 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:23:12.669155 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:23:12.669155 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:23:12.669155 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:23:12.669155 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:23:12.669155 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:23:12.669155 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:23:12.669155 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:23:12.669155 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:23:12.669155 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:23:12.669155 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:23:12.669155 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:23:12.669155 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:23:12.669155 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:23:12.669155 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:23:12.669472 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:23:12.669472 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:23:12.669472 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:12.669472 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:23:12.669966 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:12.669966 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:23:12.669966 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:23:12.669966 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:23:12.669966 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:23:12.669966 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:23:12.669966 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:23:12.669966 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:23:12.669966 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:23:12.669966 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:23:12.669966 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:23:12.669966 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:23:12.669966 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:23:12.669966 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:23:12.669966 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:23:12.669966 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:23:12.669966 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 15 10:23:12.670383 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 15 10:23:12.670383 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 15 10:23:12.670383 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 15 10:23:12.670383 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:23:12.670383 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:23:12.670383 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:23:12.670383 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:23:12.670383 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:23:12.670383 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:23:12.670383 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:23:12.670383 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:23:12.670383 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:23:12.670383 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:23:12.670383 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:23:12.670383 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:23:12.670383 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:23:12.670383 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:23:12.670383 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:23:12.670383 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:23:12.670958 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:23:12.670958 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:23:12.670958 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:12.670958 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics May 15 10:23:12.672755 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105458) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 15 10:23:12.673003 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 15 10:23:12.673112 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105458) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 15 10:23:12.673282 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105458) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 15 10:23:12.673648 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:23:12.674075 np0000174319 neutron-ovn-maintenance-worker[105458]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105458) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 15 10:23:12.674261 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:12.674983 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Traceback (most recent call last): May 15 10:23:12.674983 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:23:12.674983 np0000174319 neutron-ovn-maintenance-worker[105458]: txn.results.put(txn.do_commit()) May 15 10:23:12.674983 np0000174319 neutron-ovn-maintenance-worker[105458]: ^^^^^^^^^^^^^^^ May 15 10:23:12.674983 np0000174319 neutron-ovn-maintenance-worker[105458]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:23:12.674983 np0000174319 neutron-ovn-maintenance-worker[105458]: raise RuntimeError(msg) May 15 10:23:12.674983 np0000174319 neutron-ovn-maintenance-worker[105458]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:12.674983 np0000174319 neutron-ovn-maintenance-worker[105458]: May 15 10:23:12.675885 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:12.675885 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 15 10:23:12.675885 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:23:12.675885 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 15 10:23:12.675885 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 15 10:23:12.675885 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:23:12.675885 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 15 10:23:12.675885 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:23:12.675885 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:23:12.675885 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:23:12.675885 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 15 10:23:12.675885 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:23:12.676203 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 15 10:23:12.676203 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:23:12.676203 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 15 10:23:12.676203 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:23:12.676203 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 15 10:23:12.676203 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 15 10:23:12.676203 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:23:12.676203 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 15 10:23:12.676203 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:23:12.676203 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 15 10:23:12.676203 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 15 10:23:12.676203 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:23:12.676203 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 15 10:23:12.676513 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:12.676513 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 15 10:23:12.676985 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics [None req-a4fa6983-97d3-4a9b-9e57-664c3a561162 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:12.676985 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics Traceback (most recent call last): May 15 10:23:12.676985 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 15 10:23:12.676985 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics work() May 15 10:23:12.676985 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 15 10:23:12.676985 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 15 10:23:12.676985 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:23:12.676985 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 15 10:23:12.676985 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(*args, **kwargs) May 15 10:23:12.676985 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:23:12.676985 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 15 10:23:12.676985 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics return f(self, *args, **kwargs) May 15 10:23:12.676985 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 15 10:23:12.676985 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 15 10:23:12.676985 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics result = f(*args, **kwargs) May 15 10:23:12.676985 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 15 10:23:12.677909 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 15 10:23:12.677909 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 15 10:23:12.677909 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:23:12.677909 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:23:12.677909 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 15 10:23:12.677909 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 15 10:23:12.677909 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 15 10:23:12.677909 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics next(self.gen) May 15 10:23:12.677909 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 15 10:23:12.677909 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics with self.create_transaction( May 15 10:23:12.677909 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 15 10:23:12.677909 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics self.result = self.commit() May 15 10:23:12.677909 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 15 10:23:12.677909 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 15 10:23:12.677909 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise result.ex May 15 10:23:12.677909 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 15 10:23:12.677909 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 15 10:23:12.677909 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 15 10:23:12.677909 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 15 10:23:12.678838 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics raise RuntimeError(msg) May 15 10:23:12.678838 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 15 10:23:12.678838 np0000174319 neutron-ovn-maintenance-worker[105458]: ERROR futurist.periodics