Public bug reported:
I am running OpenStack Kilo on Ubuntu 14.04 (Trusty). I have a separate
network node running Neutron. Sometimes after reboot the neutron-
plugin-openvswitch-agent is not started, whereas the neutron-dhcp-agent
and the neutron-l3-agent get started. The problem is that sometimes (not
always) the neutron-ovs-cleanup fails to start with the following
exceptions:
2015-08-17 06:05:07.091 1470 ERROR neutron.agent.ovsdb.impl_vsctl [-] Unable to
execute ['ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--',
'list-br'].
2015-08-17 06:05:07.091 1470 TRACE neutron.agent.ovsdb.impl_vsctl Traceback
(most recent call last):
2015-08-17 06:05:07.091 1470 TRACE neutron.agent.ovsdb.impl_vsctl File
"/usr/lib/python2.7/dist-packages/neutron/agent/ovsdb/impl_vsctl.py", line 63,
in run_vsctl
2015-08-17 06:05:07.091 1470 TRACE neutron.agent.ovsdb.impl_vsctl
log_fail_as_error=False).rstrip()
2015-08-17 06:05:07.091 1470 TRACE neutron.agent.ovsdb.impl_vsctl File
"/usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 137, in
execute
2015-08-17 06:05:07.091 1470 TRACE neutron.agent.ovsdb.impl_vsctl raise
RuntimeError(m)
2015-08-17 06:05:07.091 1470 TRACE neutron.agent.ovsdb.impl_vsctl RuntimeError:
2015-08-17 06:05:07.091 1470 TRACE neutron.agent.ovsdb.impl_vsctl Command:
['sudo', '/usr/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf',
'ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--', 'list-br']
2015-08-17 06:05:07.091 1470 TRACE neutron.agent.ovsdb.impl_vsctl Exit code: 1
2015-08-17 06:05:07.091 1470 TRACE neutron.agent.ovsdb.impl_vsctl Stdin:
2015-08-17 06:05:07.091 1470 TRACE neutron.agent.ovsdb.impl_vsctl Stdout:
2015-08-17 06:05:07.091 1470 TRACE neutron.agent.ovsdb.impl_vsctl Stderr:
ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (No
such file or directory)
2015-08-17 06:05:07.091 1470 TRACE neutron.agent.ovsdb.impl_vsctl
2015-08-17 06:05:07.091 1470 TRACE neutron.agent.ovsdb.impl_vsctl
2015-08-17 06:05:07.098 1470 CRITICAL neutron [-] RuntimeError:
Command: ['sudo', '/usr/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf',
'ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--', 'list-br']
Exit code: 1
Stdin:
Stdout:
Stderr: ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection
failed (No such file or directory)
or
2015-08-16 12:08:57.315 1820 TRACE neutron Traceback (most recent call last):
2015-08-16 12:08:57.315 1820 TRACE neutron File
"/usr/bin/neutron-ovs-cleanup", line 10, in <module>
2015-08-16 12:08:57.315 1820 TRACE neutron sys.exit(main())
2015-08-16 12:08:57.315 1820 TRACE neutron File
"/usr/lib/python2.7/dist-packages/neutron/cmd/ovs_cleanup.py", line 108, in main
2015-08-16 12:08:57.315 1820 TRACE neutron delete_neutron_ports(ports)
2015-08-16 12:08:57.315 1820 TRACE neutron File
"/usr/lib/python2.7/dist-packages/neutron/cmd/ovs_cleanup.py", line 72, in
delete_neutron_ports
2015-08-16 12:08:57.315 1820 TRACE neutron device.link.delete()
2015-08-16 12:08:57.315 1820 TRACE neutron File
"/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 293, in
delete
2015-08-16 12:08:57.315 1820 TRACE neutron self._as_root([], ('delete',
self.name))
2015-08-16 12:08:57.315 1820 TRACE neutron File
"/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 222, in
_as_root
2015-08-16 12:08:57.315 1820 TRACE neutron
use_root_namespace=use_root_namespace)
2015-08-16 12:08:57.315 1820 TRACE neutron File
"/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 69, in
_as_root
2015-08-16 12:08:57.315 1820 TRACE neutron
log_fail_as_error=self.log_fail_as_error)
2015-08-16 12:08:57.315 1820 TRACE neutron File
"/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 78, in
_execute
2015-08-16 12:08:57.315 1820 TRACE neutron
log_fail_as_error=log_fail_as_error)
2015-08-16 12:08:57.315 1820 TRACE neutron File
"/usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 137, in
execute
2015-08-16 12:08:57.315 1820 TRACE neutron raise RuntimeError(m)
2015-08-16 12:08:57.315 1820 TRACE neutron RuntimeError:
2015-08-16 12:08:57.315 1820 TRACE neutron Command: ['sudo',
'/usr/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'link',
'delete', 'qg-b731d447-d1']
2015-08-16 12:08:57.315 1820 TRACE neutron Exit code: 2
2015-08-16 12:08:57.315 1820 TRACE neutron Stdin:
2015-08-16 12:08:57.315 1820 TRACE neutron Stdout:
2015-08-16 12:08:57.315 1820 TRACE neutron Stderr: RTNETLINK answers: Operation
not supported
>From my point of view there is a race condition at startup. I have added
a "sleep 5" command to my upstart-script neutron-ovs-cleanup.conf and
the problem is gone. I don't think this a proper solution.
Has anybody an idea?
** Affects: neutron
Importance: Undecided
Status: New
** Tags: neutron-core ovs
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1485422
Title:
neutron-ovs-cleanup sometimes failed at reboot (Ubuntu 14.04)
Status in neutron:
New
Bug description:
I am running OpenStack Kilo on Ubuntu 14.04 (Trusty). I have a
separate network node running Neutron. Sometimes after reboot the
neutron-plugin-openvswitch-agent is not started, whereas the neutron-
dhcp-agent and the neutron-l3-agent get started. The problem is that
sometimes (not always) the neutron-ovs-cleanup fails to start with the
following exceptions:
2015-08-17 06:05:07.091 1470 ERROR neutron.agent.ovsdb.impl_vsctl [-] Unable
to execute ['ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--',
'list-br'].
2015-08-17 06:05:07.091 1470 TRACE neutron.agent.ovsdb.impl_vsctl Traceback
(most recent call last):
2015-08-17 06:05:07.091 1470 TRACE neutron.agent.ovsdb.impl_vsctl File
"/usr/lib/python2.7/dist-packages/neutron/agent/ovsdb/impl_vsctl.py", line 63,
in run_vsctl
2015-08-17 06:05:07.091 1470 TRACE neutron.agent.ovsdb.impl_vsctl
log_fail_as_error=False).rstrip()
2015-08-17 06:05:07.091 1470 TRACE neutron.agent.ovsdb.impl_vsctl File
"/usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 137, in
execute
2015-08-17 06:05:07.091 1470 TRACE neutron.agent.ovsdb.impl_vsctl raise
RuntimeError(m)
2015-08-17 06:05:07.091 1470 TRACE neutron.agent.ovsdb.impl_vsctl
RuntimeError:
2015-08-17 06:05:07.091 1470 TRACE neutron.agent.ovsdb.impl_vsctl Command:
['sudo', '/usr/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf',
'ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--', 'list-br']
2015-08-17 06:05:07.091 1470 TRACE neutron.agent.ovsdb.impl_vsctl Exit code: 1
2015-08-17 06:05:07.091 1470 TRACE neutron.agent.ovsdb.impl_vsctl Stdin:
2015-08-17 06:05:07.091 1470 TRACE neutron.agent.ovsdb.impl_vsctl Stdout:
2015-08-17 06:05:07.091 1470 TRACE neutron.agent.ovsdb.impl_vsctl Stderr:
ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (No
such file or directory)
2015-08-17 06:05:07.091 1470 TRACE neutron.agent.ovsdb.impl_vsctl
2015-08-17 06:05:07.091 1470 TRACE neutron.agent.ovsdb.impl_vsctl
2015-08-17 06:05:07.098 1470 CRITICAL neutron [-] RuntimeError:
Command: ['sudo', '/usr/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf',
'ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--', 'list-br']
Exit code: 1
Stdin:
Stdout:
Stderr: ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection
failed (No such file or directory)
or
2015-08-16 12:08:57.315 1820 TRACE neutron Traceback (most recent call last):
2015-08-16 12:08:57.315 1820 TRACE neutron File
"/usr/bin/neutron-ovs-cleanup", line 10, in <module>
2015-08-16 12:08:57.315 1820 TRACE neutron sys.exit(main())
2015-08-16 12:08:57.315 1820 TRACE neutron File
"/usr/lib/python2.7/dist-packages/neutron/cmd/ovs_cleanup.py", line 108, in main
2015-08-16 12:08:57.315 1820 TRACE neutron delete_neutron_ports(ports)
2015-08-16 12:08:57.315 1820 TRACE neutron File
"/usr/lib/python2.7/dist-packages/neutron/cmd/ovs_cleanup.py", line 72, in
delete_neutron_ports
2015-08-16 12:08:57.315 1820 TRACE neutron device.link.delete()
2015-08-16 12:08:57.315 1820 TRACE neutron File
"/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 293, in
delete
2015-08-16 12:08:57.315 1820 TRACE neutron self._as_root([], ('delete',
self.name))
2015-08-16 12:08:57.315 1820 TRACE neutron File
"/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 222, in
_as_root
2015-08-16 12:08:57.315 1820 TRACE neutron
use_root_namespace=use_root_namespace)
2015-08-16 12:08:57.315 1820 TRACE neutron File
"/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 69, in
_as_root
2015-08-16 12:08:57.315 1820 TRACE neutron
log_fail_as_error=self.log_fail_as_error)
2015-08-16 12:08:57.315 1820 TRACE neutron File
"/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 78, in
_execute
2015-08-16 12:08:57.315 1820 TRACE neutron
log_fail_as_error=log_fail_as_error)
2015-08-16 12:08:57.315 1820 TRACE neutron File
"/usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 137, in
execute
2015-08-16 12:08:57.315 1820 TRACE neutron raise RuntimeError(m)
2015-08-16 12:08:57.315 1820 TRACE neutron RuntimeError:
2015-08-16 12:08:57.315 1820 TRACE neutron Command: ['sudo',
'/usr/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'link',
'delete', 'qg-b731d447-d1']
2015-08-16 12:08:57.315 1820 TRACE neutron Exit code: 2
2015-08-16 12:08:57.315 1820 TRACE neutron Stdin:
2015-08-16 12:08:57.315 1820 TRACE neutron Stdout:
2015-08-16 12:08:57.315 1820 TRACE neutron Stderr: RTNETLINK answers:
Operation not supported
From my point of view there is a race condition at startup. I have
added a "sleep 5" command to my upstart-script neutron-ovs-
cleanup.conf and the problem is gone. I don't think this a proper
solution.
Has anybody an idea?
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1485422/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
