Public bug reported:
FIP Namespace add/delete race conditon seen in DVR router log. This might cause
the FIP functionality to fail.
>From the trace log it seems when this happens, a bunch of tests related to FIP
>functionality fails with SSH Timeout waiting for reply.
Here is the output of the trace that kinds of shows the race condition.
Exit code: 0
execute /opt/stack/new/neutron/neutron/agent/linux/utils.py:156
2015-09-29 21:10:33.433 7884 DEBUG neutron.agent.l3.dvr_local_router [-]
Removed last floatingip, so requesting the server to delete Floatingip Agent
Gateway port:{u'allowed_address_pairs': [], u'extra_dhcp_opts': [],
u'device_owner': u'network:floatingip_agent_gateway', u'port_security_enabled':
False, u'binding:profile': {}, u'fixed_ips': [{u'subnet_id':
u'362e9033-db93-4193-9413-1073215ab326', u'prefixlen': 24, u'ip_address':
u'172.24.5.9'}, {u'subnet_id': u'feb3aa76-53b1-4d4e-b136-412c747ffd30',
u'prefixlen': 64, u'ip_address': u'2001:db8::a'}], u'id':
u'044a8e2f-00eb-4231-b526-13cb46dcc42f', u'security_groups': [],
u'binding:vif_details': {u'port_filter': True, u'ovs_hybrid_plug': True},
u'binding:vif_type': u'ovs', u'mac_address': u'fa:16:3e:7a:a6:85', u'status':
u'DOWN', u'subnets': [{u'ipv6_ra_mode': None, u'cidr': u'2001:db8::/64',
u'gateway_ip': u'2001:db8::2', u'id': u'feb3aa76-53b1-4d4e-b136-412c747ffd30',
u'subnetpool_id': None}, {u'ipv6_ra_mode': None, u'cidr': u'172.24
.5.0/24', u'gateway_ip': u'172.24.5.1', u'id':
u'362e9033-db93-4193-9413-1073215ab326', u'subnetpool_id': None}],
u'binding:host_id': u'devstack-trusty-hpcloud-b5-5153724', u'dns_assignment':
[{u'hostname': u'host-172-24-5-9', u'ip_address': u'172.24.5.9', u'fqdn':
u'host-172-24-5-9.openstacklocal.'}, {u'hostname': u'host-2001-db8--a',
u'ip_address': u'2001:db8::a', u'fqdn': u'host-2001-db8--a.openstacklocal.'}],
u'device_id': u'646bb18b-da52-4ead-a635-012c72c1ccf1', u'name': u'',
u'admin_state_up': True, u'network_id':
u'31689320-95d7-44f9-932a-cc82c1bca2b4', u'dns_name': u'',
u'binding:vnic_type': u'normal', u'tenant_id': u'', u'extra_subnets': []}
floating_ip_removed_dist
/opt/stack/new/neutron/neutron/agent/l3/dvr_local_router.py:148
2015-09-29 21:10:34.031 7884 DEBUG neutron.agent.linux.utils [-] Running
command (rootwrap daemon): ['ip', 'netns', 'delete',
'fip-31689320-95d7-44f9-932a-cc82c1bca2b4'] execute_rootwrap_daemon
/opt/stack/new/neutron/neutron/agent/linux/utils.py:101
2015-09-29 21:10:34.043 DEBUG neutron.agent.l3.dvr_local_router
[req-33413b07-784c-469e-8a35-0e20312a157e None None] FloatingIP agent gateway
port received from the plugin: {u'allowed_address_pairs': [],
u'extra_dhcp_opts': [], u'device_owner': u'network:floatingip_agent_gateway',
u'port_security_enabled': False, u'binding:profile': {}, u'fixed_ips':
[{u'subnet_id': u'362e9033-db93-4193-9413-1073215ab326', u'prefixlen': 24,
u'ip_address': u'172.24.5.9'}, {u'subnet_id':
u'feb3aa76-53b1-4d4e-b136-412c747ffd30', u'prefixlen': 64, u'ip_address':
u'2001:db8::a'}], u'id': u'044a8e2f-00eb-4231-b526-13cb46dcc42f',
u'security_groups': [], u'binding:vif_details': {u'port_filter': True,
u'ovs_hybrid_plug': True}, u'binding:vif_type': u'ovs', u'mac_address':
u'fa:16:3e:7a:a6:85', u'status': u'ACTIVE', u'subnets': [{u'ipv6_ra_mode':
None, u'cidr': u'172.24.5.0/24', u'gateway_ip': u'172.24.5.1', u'id':
u'362e9033-db93-4193-9413-1073215ab326', u'subnetpool_id': None},
{u'ipv6_ra_mode': None, u'cidr
': u'2001:db8::/64', u'gateway_ip': u'2001:db8::2', u'id':
u'feb3aa76-53b1-4d4e-b136-412c747ffd30', u'subnetpool_id': None}],
u'binding:host_id': u'devstack-trusty-hpcloud-b5-5153724', u'dns_assignment':
[{u'hostname': u'host-172-24-5-9', u'ip_address': u'172.24.5.9', u'fqdn':
u'host-172-24-5-9.openstacklocal.'}, {u'hostname': u'host-2001-db8--a',
u'ip_address': u'2001:db8::a', u'fqdn': u'host-2001-db8--a.openstacklocal.'}],
u'device_id': u'646bb18b-da52-4ead-a635-012c72c1ccf1', u'name': u'',
u'admin_state_up': True, u'network_id':
u'31689320-95d7-44f9-932a-cc82c1bca2b4', u'dns_name': u'',
u'binding:vnic_type': u'normal', u'tenant_id': u'', u'extra_subnets': []}
create_dvr_fip_interfaces
/opt/stack/new/neutron/neutron/agent/l3/dvr_local_router.py:427
2015-09-29 21:10:34.043 DEBUG neutron.agent.l3.dvr_fip_ns
[req-33413b07-784c-469e-8a35-0e20312a157e None None] add
fip-namespace(fip-31689320-95d7-44f9-932a-cc82c1bca2b4) create
/opt/stack/new/neutron/neutron/agent/l3/dvr_fip_ns.py:133
Exit code: 0
execute /opt/stack/new/neutron/neutron/agent/linux/utils.py:156
2015-09-29 21:10:34.053 DEBUG neutron.agent.linux.utils
[req-33413b07-784c-469e-8a35-0e20312a157e None None] Running command (rootwrap
daemon): ['ip', 'netns', 'exec', 'fip-31689320-95d7-44f9-932a-cc82c1bca2b4',
'sysctl', '-w', 'net.ipv4.ip_forward=1'] execute_rootwrap_daemon
/opt/stack/new/neutron/neutron/agent/linux/utils.py:101
2015-09-29 21:10:34.084 ERROR neutron.agent.linux.utils
[req-33413b07-784c-469e-8a35-0e20312a157e None None]
Command: ['ip', 'netns', 'exec', 'fip-31689320-95d7-44f9-932a-cc82c1bca2b4',
'sysctl', '-w', 'net.ipv4.ip_forward=1']
Exit code: 1
Stdin:
Stdout:
Stderr: seting the network namespace "fip-31689320-95d7-44f9-932a-cc82c1bca2b4"
failed: Invalid argument
This leads to a series of failures.
This failure is seen only in the gate.
This can be reproduced by constantly adding and deleting floatingip to a
private IP, with multiple API worker threads.
For more information you can also look at the "logstash" output below.
http://logs.openstack.org/82/228582/8/check/gate-tempest-dsvm-neutron-
dvr/9053337/logs/screen-q-l3.txt.gz?level=TRACE#_2015-09-29_21_10_34_084
** Affects: neutron
Importance: Undecided
Assignee: Swaminathan Vasudevan (swaminathan-vasudevan)
Status: In Progress
** Tags: l3-dvr-backlog
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1501873
Title:
FIP Namespace add/delete race condition seen in DVR router log
Status in neutron:
In Progress
Bug description:
FIP Namespace add/delete race conditon seen in DVR router log. This might
cause the FIP functionality to fail.
From the trace log it seems when this happens, a bunch of tests related to
FIP functionality fails with SSH Timeout waiting for reply.
Here is the output of the trace that kinds of shows the race condition.
Exit code: 0
execute /opt/stack/new/neutron/neutron/agent/linux/utils.py:156
2015-09-29 21:10:33.433 7884 DEBUG neutron.agent.l3.dvr_local_router [-]
Removed last floatingip, so requesting the server to delete Floatingip Agent
Gateway port:{u'allowed_address_pairs': [], u'extra_dhcp_opts': [],
u'device_owner': u'network:floatingip_agent_gateway', u'port_security_enabled':
False, u'binding:profile': {}, u'fixed_ips': [{u'subnet_id':
u'362e9033-db93-4193-9413-1073215ab326', u'prefixlen': 24, u'ip_address':
u'172.24.5.9'}, {u'subnet_id': u'feb3aa76-53b1-4d4e-b136-412c747ffd30',
u'prefixlen': 64, u'ip_address': u'2001:db8::a'}], u'id':
u'044a8e2f-00eb-4231-b526-13cb46dcc42f', u'security_groups': [],
u'binding:vif_details': {u'port_filter': True, u'ovs_hybrid_plug': True},
u'binding:vif_type': u'ovs', u'mac_address': u'fa:16:3e:7a:a6:85', u'status':
u'DOWN', u'subnets': [{u'ipv6_ra_mode': None, u'cidr': u'2001:db8::/64',
u'gateway_ip': u'2001:db8::2', u'id': u'feb3aa76-53b1-4d4e-b136-412c747ffd30',
u'subnetpool_id': None}, {u'ipv6_ra_mode': None, u'cidr': u'172.
24.5.0/24', u'gateway_ip': u'172.24.5.1', u'id':
u'362e9033-db93-4193-9413-1073215ab326', u'subnetpool_id': None}],
u'binding:host_id': u'devstack-trusty-hpcloud-b5-5153724', u'dns_assignment':
[{u'hostname': u'host-172-24-5-9', u'ip_address': u'172.24.5.9', u'fqdn':
u'host-172-24-5-9.openstacklocal.'}, {u'hostname': u'host-2001-db8--a',
u'ip_address': u'2001:db8::a', u'fqdn': u'host-2001-db8--a.openstacklocal.'}],
u'device_id': u'646bb18b-da52-4ead-a635-012c72c1ccf1', u'name': u'',
u'admin_state_up': True, u'network_id':
u'31689320-95d7-44f9-932a-cc82c1bca2b4', u'dns_name': u'',
u'binding:vnic_type': u'normal', u'tenant_id': u'', u'extra_subnets': []}
floating_ip_removed_dist
/opt/stack/new/neutron/neutron/agent/l3/dvr_local_router.py:148
2015-09-29 21:10:34.031 7884 DEBUG neutron.agent.linux.utils [-]
Running command (rootwrap daemon): ['ip', 'netns', 'delete',
'fip-31689320-95d7-44f9-932a-cc82c1bca2b4'] execute_rootwrap_daemon
/opt/stack/new/neutron/neutron/agent/linux/utils.py:101
2015-09-29 21:10:34.043 DEBUG neutron.agent.l3.dvr_local_router
[req-33413b07-784c-469e-8a35-0e20312a157e None None] FloatingIP agent gateway
port received from the plugin: {u'allowed_address_pairs': [],
u'extra_dhcp_opts': [], u'device_owner': u'network:floatingip_agent_gateway',
u'port_security_enabled': False, u'binding:profile': {}, u'fixed_ips':
[{u'subnet_id': u'362e9033-db93-4193-9413-1073215ab326', u'prefixlen': 24,
u'ip_address': u'172.24.5.9'}, {u'subnet_id':
u'feb3aa76-53b1-4d4e-b136-412c747ffd30', u'prefixlen': 64, u'ip_address':
u'2001:db8::a'}], u'id': u'044a8e2f-00eb-4231-b526-13cb46dcc42f',
u'security_groups': [], u'binding:vif_details': {u'port_filter': True,
u'ovs_hybrid_plug': True}, u'binding:vif_type': u'ovs', u'mac_address':
u'fa:16:3e:7a:a6:85', u'status': u'ACTIVE', u'subnets': [{u'ipv6_ra_mode':
None, u'cidr': u'172.24.5.0/24', u'gateway_ip': u'172.24.5.1', u'id':
u'362e9033-db93-4193-9413-1073215ab326', u'subnetpool_id': None},
{u'ipv6_ra_mode': None, u'ci
dr': u'2001:db8::/64', u'gateway_ip': u'2001:db8::2', u'id':
u'feb3aa76-53b1-4d4e-b136-412c747ffd30', u'subnetpool_id': None}],
u'binding:host_id': u'devstack-trusty-hpcloud-b5-5153724', u'dns_assignment':
[{u'hostname': u'host-172-24-5-9', u'ip_address': u'172.24.5.9', u'fqdn':
u'host-172-24-5-9.openstacklocal.'}, {u'hostname': u'host-2001-db8--a',
u'ip_address': u'2001:db8::a', u'fqdn': u'host-2001-db8--a.openstacklocal.'}],
u'device_id': u'646bb18b-da52-4ead-a635-012c72c1ccf1', u'name': u'',
u'admin_state_up': True, u'network_id':
u'31689320-95d7-44f9-932a-cc82c1bca2b4', u'dns_name': u'',
u'binding:vnic_type': u'normal', u'tenant_id': u'', u'extra_subnets': []}
create_dvr_fip_interfaces
/opt/stack/new/neutron/neutron/agent/l3/dvr_local_router.py:427
2015-09-29 21:10:34.043 DEBUG neutron.agent.l3.dvr_fip_ns
[req-33413b07-784c-469e-8a35-0e20312a157e None None] add
fip-namespace(fip-31689320-95d7-44f9-932a-cc82c1bca2b4) create
/opt/stack/new/neutron/neutron/agent/l3/dvr_fip_ns.py:133
Exit code: 0
execute /opt/stack/new/neutron/neutron/agent/linux/utils.py:156
2015-09-29 21:10:34.053 DEBUG neutron.agent.linux.utils
[req-33413b07-784c-469e-8a35-0e20312a157e None None] Running command (rootwrap
daemon): ['ip', 'netns', 'exec', 'fip-31689320-95d7-44f9-932a-cc82c1bca2b4',
'sysctl', '-w', 'net.ipv4.ip_forward=1'] execute_rootwrap_daemon
/opt/stack/new/neutron/neutron/agent/linux/utils.py:101
2015-09-29 21:10:34.084 ERROR neutron.agent.linux.utils
[req-33413b07-784c-469e-8a35-0e20312a157e None None]
Command: ['ip', 'netns', 'exec', 'fip-31689320-95d7-44f9-932a-cc82c1bca2b4',
'sysctl', '-w', 'net.ipv4.ip_forward=1']
Exit code: 1
Stdin:
Stdout:
Stderr: seting the network namespace
"fip-31689320-95d7-44f9-932a-cc82c1bca2b4" failed: Invalid argument
This leads to a series of failures.
This failure is seen only in the gate.
This can be reproduced by constantly adding and deleting floatingip to
a private IP, with multiple API worker threads.
For more information you can also look at the "logstash" output below.
http://logs.openstack.org/82/228582/8/check/gate-tempest-dsvm-neutron-
dvr/9053337/logs/screen-q-l3.txt.gz?level=TRACE#_2015-09-29_21_10_34_084
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1501873/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
