Public bug reported:
Environment:
ii neutron-plugin-ml2 2:7.0.3-0ubuntu1~cloud0 all
Neutron is a virtual network service for Openstack - ML2 plugin
ii neutron-server 2:7.0.3-0ubuntu1~cloud0 all
Neutron is a virtual network service for Openstack - server
:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 14.04.4 LTS
Release: 14.04
Codename: trusty
Kernel: 3.18.26-x1-64
}}}
After upgrading Kilo to Liberty (using Ubuntu Cloud Archive Packages).
Neutron server fails to start.
We have been able to trace this back to the following query.
{{{
[SQL:
u'SELECT ml2_vlan_allocations.physical_network AS
ml2_vlan_allocations_physical_network, ml2_vlan_allocations.vlan_id AS
ml2_vlan_allocations_vlan_id, ml2_vlan_
allocations.allocated AS ml2_vlan_allocations_allocated \nFROM
ml2_vlan_allocations FOR UPDATE']
}}}
It appears that neutron is locking the MySQL database, and a little
later, MySQL is dropping the lock causing the following:
{{{
neutron [-] DBError: (pymysql.err.InternalError) (1205, u'Lock wait timeout
exceeded; try restarting transaction')
}}}
Example of where Neutron Server is stuck starting up:
{{{
2016-05-16 12:39:07.859 13463 INFO neutron.manager [-] Loading core plugin: ml2
2016-05-16 12:39:07.960 13463 INFO neutron.plugins.ml2.managers [-] Configured
type driver names: ['flat', 'gre', 'vlan']
2016-05-16 12:39:07.963 13463 INFO neutron.plugins.ml2.drivers.type_flat [-]
Allowable flat physical_network names: ['external']
2016-05-16 12:39:07.966 13463 INFO neutron.plugins.ml2.drivers.type_vlan [-]
Network VLAN ranges: {'vlan': [(300, 4000)]}
2016-05-16 12:39:07.972 13463 INFO neutron.plugins.ml2.managers [-] Loaded type
driver names: ['flat', 'vlan', 'gre']
2016-05-16 12:39:07.973 13463 INFO neutron.plugins.ml2.managers [-] Registered
types: ['flat', 'vlan', 'gre']
2016-05-16 12:39:07.973 13463 INFO neutron.plugins.ml2.managers [-] Tenant
network_types: ['gre']
2016-05-16 12:39:07.974 13463 INFO neutron.plugins.ml2.managers [-] Configured
extension driver names: []
2016-05-16 12:39:07.974 13463 INFO neutron.plugins.ml2.managers [-] Loaded
extension driver names: []
2016-05-16 12:39:07.975 13463 INFO neutron.plugins.ml2.managers [-] Registered
extension drivers: []
2016-05-16 12:39:07.975 13463 INFO neutron.plugins.ml2.managers [-] Configured
mechanism driver names: ['openvswitch']
2016-05-16 12:39:07.977 13463 INFO neutron.plugins.ml2.managers [-] Loaded
mechanism driver names: ['openvswitch']
2016-05-16 12:39:07.978 13463 INFO neutron.plugins.ml2.managers [-] Registered
mechanism drivers: ['openvswitch']
2016-05-16 12:39:08.062 13463 DEBUG neutron.callbacks.manager [-] Subscribe:
<bound method Ml2Plugin.validate_network_rbac_policy_change of
<neutron.plugins.ml2.plugin.Ml2Plugin object at 0x7ff208835dd0>> rbac-policy
before_create subscribe
/usr/lib/python2.7/dist-packages/neutron/callbacks/manager.py:41
2016-05-16 12:39:08.062 13463 DEBUG neutron.callbacks.manager [-] Subscribe:
<bound method Ml2Plugin.validate_network_rbac_policy_change of
<neutron.plugins.ml2.plugin.Ml2Plugin object at 0x7ff208835dd0>> rbac-policy
before_update subscribe
/usr/lib/python2.7/dist-packages/neutron/callbacks/manager.py:41
2016-05-16 12:39:08.063 13463 DEBUG neutron.callbacks.manager [-] Subscribe:
<bound method Ml2Plugin.validate_network_rbac_policy_change of
<neutron.plugins.ml2.plugin.Ml2Plugin object at 0x7ff208835dd0>> rbac-policy
before_delete subscribe
/usr/lib/python2.7/dist-packages/neutron/callbacks/manager.py:41
2016-05-16 12:39:08.064 13463 INFO neutron.plugins.ml2.managers [-]
Initializing driver for type 'flat'
2016-05-16 12:39:08.064 13463 INFO neutron.plugins.ml2.drivers.type_flat [-]
ML2 FlatTypeDriver initialization complete
2016-05-16 12:39:08.065 13463 INFO neutron.plugins.ml2.managers [-]
Initializing driver for type 'vlan'
2016-05-16 12:39:08.115 13463 DEBUG oslo_db.sqlalchemy.engines [-] MySQL server
mode set to
STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION
_check_effective_sql_mode
/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/engines.py:256
}}}
We are able to use the following to populate a fresh database, then
neutron will start.
{{{
su -s /bin/sh -c "neutron-db-manage --config-file /etc/neutron/neutron.conf
--config-file /etc/neutron/plugins/ml2/ml2_conf.ini upgrade head" neutron
}}}
However as you can imagine its quite important that we are able to
upgrade the current database!
Appreciate any help, or links to previous related bugs that may improve
our situation.
** Affects: neutron
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1582224
Title:
Post liberty upgrade neutron-server wont start - seemingly related to
ml2_vlan_allocations table
Status in neutron:
New
Bug description:
Environment:
ii neutron-plugin-ml2 2:7.0.3-0ubuntu1~cloud0
all Neutron is a virtual network service for Openstack - ML2 plugin
ii neutron-server 2:7.0.3-0ubuntu1~cloud0
all Neutron is a virtual network service for Openstack - server
:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 14.04.4 LTS
Release: 14.04
Codename: trusty
Kernel: 3.18.26-x1-64
}}}
After upgrading Kilo to Liberty (using Ubuntu Cloud Archive Packages).
Neutron server fails to start.
We have been able to trace this back to the following query.
{{{
[SQL:
u'SELECT ml2_vlan_allocations.physical_network AS
ml2_vlan_allocations_physical_network, ml2_vlan_allocations.vlan_id AS
ml2_vlan_allocations_vlan_id, ml2_vlan_
allocations.allocated AS ml2_vlan_allocations_allocated \nFROM
ml2_vlan_allocations FOR UPDATE']
}}}
It appears that neutron is locking the MySQL database, and a little
later, MySQL is dropping the lock causing the following:
{{{
neutron [-] DBError: (pymysql.err.InternalError) (1205, u'Lock wait timeout
exceeded; try restarting transaction')
}}}
Example of where Neutron Server is stuck starting up:
{{{
2016-05-16 12:39:07.859 13463 INFO neutron.manager [-] Loading core plugin:
ml2
2016-05-16 12:39:07.960 13463 INFO neutron.plugins.ml2.managers [-]
Configured type driver names: ['flat', 'gre', 'vlan']
2016-05-16 12:39:07.963 13463 INFO neutron.plugins.ml2.drivers.type_flat [-]
Allowable flat physical_network names: ['external']
2016-05-16 12:39:07.966 13463 INFO neutron.plugins.ml2.drivers.type_vlan [-]
Network VLAN ranges: {'vlan': [(300, 4000)]}
2016-05-16 12:39:07.972 13463 INFO neutron.plugins.ml2.managers [-] Loaded
type driver names: ['flat', 'vlan', 'gre']
2016-05-16 12:39:07.973 13463 INFO neutron.plugins.ml2.managers [-]
Registered types: ['flat', 'vlan', 'gre']
2016-05-16 12:39:07.973 13463 INFO neutron.plugins.ml2.managers [-] Tenant
network_types: ['gre']
2016-05-16 12:39:07.974 13463 INFO neutron.plugins.ml2.managers [-]
Configured extension driver names: []
2016-05-16 12:39:07.974 13463 INFO neutron.plugins.ml2.managers [-] Loaded
extension driver names: []
2016-05-16 12:39:07.975 13463 INFO neutron.plugins.ml2.managers [-]
Registered extension drivers: []
2016-05-16 12:39:07.975 13463 INFO neutron.plugins.ml2.managers [-]
Configured mechanism driver names: ['openvswitch']
2016-05-16 12:39:07.977 13463 INFO neutron.plugins.ml2.managers [-] Loaded
mechanism driver names: ['openvswitch']
2016-05-16 12:39:07.978 13463 INFO neutron.plugins.ml2.managers [-]
Registered mechanism drivers: ['openvswitch']
2016-05-16 12:39:08.062 13463 DEBUG neutron.callbacks.manager [-] Subscribe:
<bound method Ml2Plugin.validate_network_rbac_policy_change of
<neutron.plugins.ml2.plugin.Ml2Plugin object at 0x7ff208835dd0>> rbac-policy
before_create subscribe
/usr/lib/python2.7/dist-packages/neutron/callbacks/manager.py:41
2016-05-16 12:39:08.062 13463 DEBUG neutron.callbacks.manager [-] Subscribe:
<bound method Ml2Plugin.validate_network_rbac_policy_change of
<neutron.plugins.ml2.plugin.Ml2Plugin object at 0x7ff208835dd0>> rbac-policy
before_update subscribe
/usr/lib/python2.7/dist-packages/neutron/callbacks/manager.py:41
2016-05-16 12:39:08.063 13463 DEBUG neutron.callbacks.manager [-] Subscribe:
<bound method Ml2Plugin.validate_network_rbac_policy_change of
<neutron.plugins.ml2.plugin.Ml2Plugin object at 0x7ff208835dd0>> rbac-policy
before_delete subscribe
/usr/lib/python2.7/dist-packages/neutron/callbacks/manager.py:41
2016-05-16 12:39:08.064 13463 INFO neutron.plugins.ml2.managers [-]
Initializing driver for type 'flat'
2016-05-16 12:39:08.064 13463 INFO neutron.plugins.ml2.drivers.type_flat [-]
ML2 FlatTypeDriver initialization complete
2016-05-16 12:39:08.065 13463 INFO neutron.plugins.ml2.managers [-]
Initializing driver for type 'vlan'
2016-05-16 12:39:08.115 13463 DEBUG oslo_db.sqlalchemy.engines [-] MySQL
server mode set to
STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION
_check_effective_sql_mode
/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/engines.py:256
}}}
We are able to use the following to populate a fresh database, then
neutron will start.
{{{
su -s /bin/sh -c "neutron-db-manage --config-file /etc/neutron/neutron.conf
--config-file /etc/neutron/plugins/ml2/ml2_conf.ini upgrade head" neutron
}}}
However as you can imagine its quite important that we are able to
upgrade the current database!
Appreciate any help, or links to previous related bugs that may
improve our situation.
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1582224/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : [email protected]
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
