All,
I’m noticing the following Neutron server error “failed scheduling network”
when a user attempts to startup multiple instances at one time. In this case,
a number of instances fail to configure their network access.
I suspect this is a scaling issue, as we’re running a relatively small
Openstack testbed and have multiple services running on the same controller
node. Neutron is configure with a single DHCP agent. Instances are being
brought up on a flat network that maps directly to our campus backbone. A
sample of the neutron server log file shows the following:
NEUTRON: 2014-07-15 14:01:56 WARNING [neutron.db.agentschedulers_db] Fail
scheduling network {'status': u'A
CTIVE', 'subnets': [u'4df48f6b-8d6e-4a5a-b922-d3503add14a7',
u'b90d7486-7ecb-425c-b9ba-180ba4c936df'], 'name
': u'campus-net', 'provider:physical_network': u'default', 'admin_state_up':
True, 'tenant_id': u'4414273897
e44c35ab99cd73a3996787', 'provider:network_type': u'flat', 'router:external':
False, 'shared': False, 'id':
u'a9672c56-5491-4052-aa55-c25507ba4be7', 'provider:segmentation_id': None}
NEUTRON: 2014-07-15 14:01:56 WARNING [neutron.db.agentschedulers_db] Fail
scheduling network {'status': u'A
CTIVE', 'subnets': [u'4df48f6b-8d6e-4a5a-b922-d3503add14a7',
u'b90d7486-7ecb-425c-b9ba-180ba4c936df'], 'name
': u'campus-net', 'provider:physical_network': u'default', 'admin_state_up':
True, 'tenant_id': u'4414273897
e44c35ab99cd73a3996787', 'provider:network_type': u'flat', 'router:external':
False, 'shared': False, 'id':
u'a9672c56-5491-4052-aa55-c25507ba4be7', 'provider:segmentation_id': None}
NEUTRON: 2014-07-15 14:01:57 WARNING [neutron.db.agentschedulers_db] Fail
scheduling network {'status': u'A
CTIVE', 'subnets': [u'4df48f6b-8d6e-4a5a-b922-d3503add14a7',
u'b90d7486-7ecb-425c-b9ba-180ba4c936df'], 'name
': u'campus-net', 'provider:physical_network': u'default', 'admin_state_up':
True, 'tenant_id': u'4414273897
e44c35ab99cd73a3996787', 'provider:network_type': u'flat', 'router:external':
False, 'shared': False, 'id':
u'a9672c56-5491-4052-aa55-c25507ba4be7', 'provider:segmentation_id': None}
NEUTRON: 2014-07-15 14:01:57 WARNING [neutron.db.agentschedulers_db] Fail
scheduling network {'status': u'A
CTIVE', 'subnets': [u'4df48f6b-8d6e-4a5a-b922-d3503add14a7',
u'b90d7486-7ecb-425c-b9ba-180ba4c936df'], 'name
': u'campus-net', 'provider:physical_network': u'default', 'admin_state_up':
True, 'tenant_id': u'4414273897
e44c35ab99cd73a3996787', 'provider:network_type': u'flat', 'router:external':
False, 'shared': False, 'id':
u'a9672c56-5491-4052-aa55-c25507ba4be7', 'provider:segmentation_id': None}
NEUTRON: 2014-07-15 14:01:58 WARNING [neutron.db.agentschedulers_db] Fail
scheduling network {'status': u'A
CTIVE', 'subnets': [u'4df48f6b-8d6e-4a5a-b922-d3503add14a7',
u'b90d7486-7ecb-425c-b9ba-180ba4c936df'], 'name
': u'campus-net', 'provider:physical_network': u'default', 'admin_state_up':
True, 'tenant_id': u'4414273897
e44c35ab99cd73a3996787', 'provider:network_type': u'flat', 'router:external':
False, 'shared': False, 'id':
u'a9672c56-5491-4052-aa55-c25507ba4be7', 'provider:segmentation_id': None}
NEUTRON: 2014-07-15 14:05:09 WARNING [neutron.db.agentschedulers_db] Fail
scheduling network {'status': u'A
CTIVE', 'subnets': [u'4df48f6b-8d6e-4a5a-b922-d3503add14a7',
u'b90d7486-7ecb-425c-b9ba-180ba4c936df'], 'name
': u'campus-net', 'provider:physical_network': u'default', 'admin_state_up':
True, 'tenant_id': u'4414273897
e44c35ab99cd73a3996787', 'provider:network_type': u'flat', 'router:external':
False, 'shared': False, 'id':
u'a9672c56-5491-4052-aa55-c25507ba4be7', 'provider:segmentation_id': None}
etc., etc., etc.
Question. Is this a simple scaling issue, where the DHCP agent is unable to
respond to scheduling requests from the neutron server? Or is my understanding
naive, and something else is going on? The corresponding DHCP log for the same
time period, shows the following:
NEUTRON: 2014-07-15 14:01:56 DEBUG [neutron.agent.linux.utils] Running
command: ['sudo', '/usr/bin/neutro
n-rootwrap', '/etc/neutron/rootwrap.conf', 'kill', '-HUP', '46267']
NEUTRON: 2014-07-15 14:01:57 DEBUG [neutron.agent.linux.utils]
Command: ['sudo', '/usr/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf',
'kill', '-HUP', '46267']
Exit code: 0
Stdout: ''
Stderr: ''
NEUTRON: 2014-07-15 14:01:57 DEBUG [neutron.agent.linux.dhcp] Reloading
allocations for network: a9672c56
-5491-4052-aa55-c25507ba4be7
NEUTRON: 2014-07-15 14:01:57 DEBUG [neutron.openstack.common.rpc.amqp]
Making synchronous call on q-plugi
n ...
NEUTRON: 2014-07-15 14:01:57 DEBUG [neutron.openstack.common.rpc.amqp]
MSG_ID is 6be3406bc1c747e2890cd53d
c24f5e4d
NEUTRON: 2014-07-15 14:01:57 DEBUG [neutron.openstack.common.rpc.amqp]
UNIQUE_ID is db708afb7b6848be9f07e
395e7c27cad.
NEUTRON: 2014-07-15 14:01:57 DEBUG [neutron.openstack.common.rpc.amqp]
Making asynchronous cast on q-plug
in...
NEUTRON: 2014-07-15 14:01:57 DEBUG [neutron.openstack.common.rpc.amqp]
UNIQUE_ID is cb882dab0a6e4c12b1145
189f45a345f.
NEUTRON: 2014-07-15 14:01:57 DEBUG [amqp] Closed channel #1
NEUTRON: 2014-07-15 14:01:57 DEBUG [amqp] using channel_id: 1
NEUTRON: 2014-07-15 14:01:57 DEBUG [amqp] Channel open
NEUTRON: 2014-07-15 14:01:57 DEBUG [amqp] Closed channel #1
NEUTRON: 2014-07-15 14:01:57 DEBUG [amqp] using channel_id: 1
NEUTRON: 2014-07-15 14:01:57 DEBUG [amqp] Channel open
NEUTRON: 2014-07-15 14:01:57 DEBUG [neutron.openstack.common.rpc.amqp]
received {u'_context_roles': [u'ad
min'], u'_context_read_deleted': u'no', u'_context_tenant_id':
u'f59f41096eb142ccbe72e7cf6702518c', u'args':
{u'payload': {u'port': {u'status': u'DOWN', u'binding:host_id': u'cn6',
u'name': u'', u'allowed_address_pai
rs': [], u'admin_state_up': True, u'network_id':
u'a9672c56-5491-4052-aa55-c25507ba4be7', u'tenant_id': u'44
14273897e44c35ab99cd73a3996787', u'binding:vif_type': u'ovs', u'device_owner':
u'compute:nova', u'binding:ca
pabilities': {u'port_filter': True}, u'mac_address': u'fa:16:3e:4f:88:54',
u'fixed_ips': [{u'subnet_id': u'4
df48f6b-8d6e-4a5a-b922-d3503add14a7', u'ip_address': u'173.23.187.15'}], u'id':
u'1924e8b5-ea22-4474-adc1-fc
3194401cc8', u'security_groups': [u'2f8ef352-adec-48f5-ba17-bb55f85de7de'],
u'device_id': u'7bf73c30-d949-4f
34-886f-68c9a503e963'}}}, u'namespace': None, u'_unique_id':
u'a7d139060a874c3b832af33491d8c469', u'_context
_is_admin': True, u'version': u'1.0', u'_context_project_id':
u'f59f41096eb142ccbe72e7cf6702518c', u'_contex
t_timestamp': u'2014-07-15 19:01:54.484397', u'_context_user_id':
u'6f0f1783946348f7b01881501e2f5b9c', u'met
hod': u'port_create_end'}
NEUTRON: 2014-07-15 14:01:57 DEBUG [neutron.openstack.common.rpc.amqp]
unpacked context: {'user_id': u'6f
0f1783946348f7b01881501e2f5b9c', 'roles': [u'admin'], 'tenant_id':
u'f59f41096eb142ccbe72e7cf6702518c', 'is_
admin': True, 'timestamp': u'2014-07-15 19:01:54.484397', 'project_id':
u'f59f41096eb142ccbe72e7cf6702518c',
'read_deleted': u'no'}
NEUTRON: 2014-07-15 14:01:57 DEBUG [neutron.openstack.common.rpc.amqp]
received {u'_context_roles': [u'ad
min'], u'_context_read_deleted': u'no', u'_context_tenant_id':
u'f59f41096eb142ccbe72e7cf6702518c', u'args':
{u'payload': {u'port': {u'status': u'DOWN', u'binding:host_id': u'cn2',
u'name': u'', u'allowed_address_pai
rs': [], u'admin_state_up': True, u'network_id':
u'a9672c56-5491-4052-aa55-c25507ba4be7', u'tenant_id': u'44
14273897e44c35ab99cd73a3996787', u'binding:vif_type': u'ovs', u'device_owner':
u'compute:nova', u'binding:ca
pabilities': {u'port_filter': True}, u'mac_address': u'fa:16:3e:4d:5b:33',
u'fixed_ips': [{u'subnet_id': u'4
df48f6b-8d6e-4a5a-b922-d3503add14a7', u'ip_address': u'173.23.187.16'}], u'id':
u'02438722-85a7-4d49-b74b-66
61650bbcd0', u'security_groups': [u'2f8ef352-adec-48f5-ba17-bb55f85de7de'],
u'device_id': u'242437ce-d346-43
d6-b6c9-41777eb8d669'}}}, u'namespace': None, u'_unique_id':
u'5a162a32393f4506a8f9d42b963c44b2', u'_context
_is_admin': True, u'version': u'1.0', u'_context_project_id':
u'f59f41096eb142ccbe72e7cf6702518c', u'_contex
t_timestamp': u'2014-07-15 19:01:54.490335', u'_context_user_id':
u'6f0f1783946348f7b01881501e2f5b9c', u'met
hod': u'port_create_end'}
NEUTRON: 2014-07-15 14:01:57 DEBUG [neutron.openstack.common.rpc.amqp]
unpacked context: {'user_id': u'6f
0f1783946348f7b01881501e2f5b9c', 'roles': [u'admin'], 'tenant_id':
u'f59f41096eb142ccbe72e7cf6702518c', 'is_
admin
Do we have some type of a race condition going on here?
Thoughts?
/ross
--
Ross Lillie
Distinguished Member of Technical Staff
Motorola Solutions, Inc.
motorolasolutions.com<http://motorolasolutions.com>
O: +1.847.576.0012
M: +1.847.980.2241
E: ross.lil...@motorolasolutions.com<mailto:ross.lil...@motorolasolutions.com>
[cid:75FF7557-F6FD-4406-A0BB-5CBD9BE3780A@comm.mot.com]
_______________________________________________
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
