Gary and I just chatted on IRC about this. Nova makes no assumptions about whether networks are created before or after nova-network is started, and a nova-network restart would trigger the same behavior anyway, so it seems best to just try and fix the issue with the LB interface-driver directly.
dan On Thu, May 17, 2012 at 11:27 AM, Gary Kotton <gkot...@redhat.com> wrote: > ** > Hi, > After fix https://review.openstack.org/#/c/7550/ and moving the "network > create" command to after the starting of the network service the > Linuxbridge plugin work. > Dan would it be possible that you move the creation of the "nova network" > to after the start of nova network in > https://review.openstack.org/#/c/7001/? > > Thanks > Gary > > On 05/17/2012 12:07 AM, Dan Wendlandt wrote: > > > > On Wed, May 16, 2012 at 2:01 PM, Sumit Naiksatam (snaiksat) < > snaik...@cisco.com> wrote: > >> The interesting data point here is that this only happens to the >> default network created when the installation is done via DevStack. For all >> network created subsequently I am not seeing this issue. I earlier thought >> that this might have something to do with the exercises script getting >> executed and possibly leaving a residue. However, in spite of not running >> the exercises script I am still seeing this issue. It seems that for some >> reason, the initialize_gateway() was getting called only when a new VM was >> being created. But here it seems to be getting called before that (thus >> leading to the creation of the bridge and the gateway devices), not sure >> why. I am just trying to figure out how I can test the fix with respect to >> the devstack setup (since this happens only on installation)… >> > > Your explanation makes sense. I think there was a recent comment to > automatically initialize all networks when nova-network is started, in case > nova-network was started on a new host (e.g., in an HA configuration) or > after a host reboot. Since devstack creates the network using nova-manage > before starting nova-network, this would trigger the behavior your > describe. For more info, see: > https://github.com/openstack/nova/commit/30c8293ea7d4c9464e0139fb488e202fd0947f52 > > I suspect you should hit this every time you re-run stack.sh, no? > > > > > >> >> >> *From:* Dan Wendlandt [mailto:d...@nicira.com] >> *Sent:* Wednesday, May 16, 2012 7:55 AM >> *To:* gkot...@redhat.com >> *Cc:* Sumit Naiksatam (snaiksat); netstack@lists.launchpad.net >> *Subject:* Re: [Netstack] Fwd: [Openstack] [devstack/quantum] >> Configuration issue >> >> >> >> >> >> On Wed, May 16, 2012 at 1:43 AM, Gary Kotton <gkot...@redhat.com> wrote: >> >> Thanks! >> As far as I recall, and I may certain be wrong here, but in the past I >> did not see the GW created until the first VM was deployed. Now I am seeing >> the GW created when the various services are started. >> >> >> >> I have noticed this as well. I'm not aware of any changes to >> QuantumManager that had this affect, though its possible someone pushed a >> change that I didn't notice. I primarily use the OVS interface-driver and >> didn't notice any issues, so I suspect Sumit is correct that we can at >> least work around this with a change to the LB interface-driver (even >> though the change that caused the issue was likely somewhere else). >> >> >> >> This is a good example of why we need Quantum integrated into devstack >> commit-gating, as it will catch this type of complex integration issues. >> Anyone have cycles to help push on this? I'm unlikely to make much >> progress on it in the next week or two due to the F-1 release. >> >> >> >> Dan >> >> >> >> >> >> >> >> Thanks >> Gary >> >> >> >> On 05/16/2012 10:54 AM, Sumit Naiksatam (snaiksat) wrote: >> >> This is probably not a devstack issue. The problem is with the LB gateway >> driver wherein the IP address is trying to be set on a bridge device which >> already has an IP address. The check for an existing bridge is being >> performed, but the IP address is being set outside that check. Ideally, >> this code should not have been invoked if the gateway was already set. >> Something seems to have changed in the QuantumManager as result of which >> this code is being invoked again. At any rate, I will fix the LB gateway >> driver, and we will not see this. >> >> >> >> Thanks, >> >> ~Sumit. >> >> >> >> *From:* netstack-bounces+snaiksat=cisco....@lists.launchpad.net [ >> mailto:netstack-bounces+snaiksat=cisco....@lists.launchpad.net<netstack-bounces+snaiksat=cisco....@lists.launchpad.net>] >> *On Behalf Of *Gary Kotton >> *Sent:* Monday, May 14, 2012 11:19 PM >> *To:* <netstack@lists.launchpad.net> <netstack@lists.launchpad.net> >> *Subject:* [Netstack] Fwd: [Openstack] [devstack/quantum] Configuration >> issue >> >> >> >> >> >> -------- Original Message -------- >> >> *Subject: * >> >> [Openstack] [devstack/quantum] Configuration issue >> >> *Date: * >> >> Tue, 15 May 2012 09:00:52 +0300 >> >> *From: * >> >> Gary Kotton <gkot...@redhat.com> <gkot...@redhat.com> >> >> *Reply-To: * >> >> gkot...@redhat.com >> >> *Organization: * >> >> Red Hat >> >> *To: * >> >> openst...@lists.launchpad.net >> >> >> >> Hi, >> >> This morning I encountered a problem (which did not happen a few days >> >> ago :)). When devstack is launched, with quantum configured, the gateway >> >> and bridge devices are created. This causes problems with quantum. >> >> >> >> For example when devstack is up and running prior to deploying an >> >> instance we have: >> >> >> >> brq744ec2f4-c0 Link encap:Ethernet HWaddr fa:16:3e:03:a6:55 >> >> inet addr:10.0.0.1 Bcast:0.0.0.0 Mask:255.255.255.0 >> >> UP BROADCAST MULTICAST MTU:1500 Metric:1 >> >> RX packets:0 errors:0 dropped:0 overruns:0 frame:0 >> >> TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 >> >> collisions:0 txqueuelen:0 >> >> RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) >> >> >> >> gw-744ec2f4-c0 Link encap:Ethernet HWaddr fa:16:3e:03:a6:55 >> >> inet addr:10.0.0.1 Bcast:10.0.0.255 Mask:255.255.255.0 >> >> UP BROADCAST MULTICAST MTU:1500 Metric:1 >> >> RX packets:0 errors:0 dropped:0 overruns:0 frame:0 >> >> TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 >> >> collisions:0 txqueuelen:500 >> >> RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) >> >> >> >> When an instance is deployed the following happens: >> >> >> >> 2012-05-15 01:59:18 DEBUG nova.utils >> >> [req-4d50ed10-46e1-406c-9074-dc45da860365 >> >> df07eec326434b25800f3ebc17202fb3 2cafe0be4d7740098a89be39ffd1b72e] >> >> Running cmd (subprocess): sudo /usr/local/bin/nova-rootwrap ip address >> >> add 10.0.0.1/24 dev brq744ec2f4-c0 from (pid=4234) execute >> >> /opt/stack/nova/nova/utils.py:178 >> >> 2012-05-15 01:59:18 DEBUG nova.utils >> >> [req-4d50ed10-46e1-406c-9074-dc45da860365 >> >> df07eec326434b25800f3ebc17202fb3 2cafe0be4d7740098a89be39ffd1b72e] >> >> Result was 254 from (pid=4234) execute /opt/stack/nova/nova/utils.py:194 >> >> 2012-05-15 01:59:18 ERROR nova.rpc.amqp >> >> [req-4d50ed10-46e1-406c-9074-dc45da860365 >> >> df07eec326434b25800f3ebc17202fb3 2cafe0be4d7740098a89be39ffd1b72e] >> >> Exception during message handling >> >> 2012-05-15 01:59:18 TRACE nova.rpc.amqp Traceback (most recent call last): >> >> 2012-05-15 01:59:18 TRACE nova.rpc.amqp File >> >> "/opt/stack/nova/nova/rpc/amqp.py", line 263, in _process_data >> >> 2012-05-15 01:59:18 TRACE nova.rpc.amqp rval = >> >> node_func(context=ctxt, **node_args) >> >> 2012-05-15 01:59:18 TRACE nova.rpc.amqp File >> >> "/opt/stack/nova/nova/network/quantum/manager.py", line 390, in >> >> allocate_for_instance >> >> 2012-05-15 01:59:18 TRACE nova.rpc.amqp network, vif_rec, >> >> network['net_tenant_id']) >> >> 2012-05-15 01:59:18 TRACE nova.rpc.amqp File >> >> "/opt/stack/nova/nova/utils.py", line 880, in inner >> >> 2012-05-15 01:59:18 TRACE nova.rpc.amqp retval = f(*args, **kwargs) >> >> 2012-05-15 01:59:18 TRACE nova.rpc.amqp File >> >> "/opt/stack/nova/nova/network/quantum/manager.py", line 501, in enable_dhcp >> >> 2012-05-15 01:59:18 TRACE nova.rpc.amqp >> >> self.l3driver.initialize_gateway(network_ref) >> >> 2012-05-15 01:59:18 TRACE nova.rpc.amqp File >> >> "/opt/stack/nova/nova/network/l3.py", line 98, in initialize_gateway >> >> 2012-05-15 01:59:18 TRACE nova.rpc.amqp >> >> gateway=(network_ref['gateway'] is not None)) >> >> 2012-05-15 01:59:18 TRACE nova.rpc.amqp File >> >> "/opt/stack/nova/nova/network/linux_net.py", line 900, in plug >> >> 2012-05-15 01:59:18 TRACE nova.rpc.amqp return >> >> _get_interface_driver().plug(network, mac_address, gateway) >> >> 2012-05-15 01:59:18 TRACE nova.rpc.amqp File >> >> "/opt/stack/nova/nova/network/linux_net.py", line 1160, in plug >> >> 2012-05-15 01:59:18 TRACE nova.rpc.amqp run_as_root=True) >> >> 2012-05-15 01:59:18 TRACE nova.rpc.amqp File >> >> "/opt/stack/nova/nova/utils.py", line 201, in execute >> >> 2012-05-15 01:59:18 TRACE nova.rpc.amqp cmd=' '.join(cmd)) >> >> 2012-05-15 01:59:18 TRACE nova.rpc.amqp ProcessExecutionError: >> >> Unexpected error while running command. >> >> 2012-05-15 01:59:18 TRACE nova.rpc.amqp Command: sudo >> >> /usr/local/bin/nova-rootwrap ip address add 10.0.0.1/24 dev brq744ec2f4-c0 >> >> 2012-05-15 01:59:18 TRACE nova.rpc.amqp Exit code: 254 >> >> 2012-05-15 01:59:18 TRACE nova.rpc.amqp Stdout: '' >> >> 2012-05-15 01:59:18 TRACE nova.rpc.amqp Stderr: 'RTNETLINK answers: File >> >> exists\n' >> >> 2012-05-15 01:59:18 TRACE nova.rpc.amqp >> >> 2012-05-15 01:59:18 ERROR nova.rpc.common >> >> [req-4d50ed10-46e1-406c-9074-dc45da860365 >> >> df07eec326434b25800f3ebc17202fb3 2cafe0be4d7740098a89be39ffd1b72e] >> >> Returning exception Unexpected error while running command. >> >> Command: sudo /usr/local/bin/nova-rootwrap ip address add 10.0.0.1/24 >> >> dev brq744ec2f4-c0 >> >> Exit code: 254 >> >> Stdout: '' >> >> Stderr: 'RTNETLINK answers: File exists\n' to caller >> >> 2012-05-15 01:59:18 ERROR nova.rpc.common >> >> [req-4d50ed10-46e1-406c-9074-dc45da860365 >> >> df07eec326434b25800f3ebc17202fb3 2cafe0be4d7740098a89be39ffd1b72e] >> >> ['Traceback (most recent call last):\n', ' File >> >> "/opt/stack/nova/nova/rpc/amqp.py", line 263, in _process_data\n rval >> >> = node_func(context=ctxt, **node_args)\n', ' File >> >> "/opt/stack/nova/nova/network/quantum/manager.py", line 390, in >> >> allocate_for_instance\n network, vif_rec, >> >> network[\'net_tenant_id\'])\n', ' File "/opt/stack/nova/nova/utils.py", >> >> line 880, in inner\n retval = f(*args, **kwargs)\n', ' File >> >> "/opt/stack/nova/nova/network/quantum/manager.py", line 501, in >> >> enable_dhcp\n self.l3driver.initialize_gateway(network_ref)\n', ' >> >> File "/opt/stack/nova/nova/network/l3.py", line 98, in >> >> initialize_gateway\n gateway=(network_ref[\'gateway\'] is not >> >> None))\n', ' File "/opt/stack/nova/nova/network/linux_net.py", line >> >> 900, in plug\n return _get_interface_driver().plug(network, >> >> mac_address, gateway)\n', ' File >> >> "/opt/stack/nova/nova/network/linux_net.py", line 1160, in plug\n >> >> run_as_root=True)\n', ' File "/opt/stack/nova/nova/utils.py", line 201, >> >> in execute\n cmd=\' \'.join(cmd))\n', "ProcessExecutionError: >> >> Unexpected error while running command.\nCommand: sudo >> >> /usr/local/bin/nova-rootwrap ip address add 10.0.0.1/24 dev >> >> brq744ec2f4-c0\nExit code: 254\nStdout: ''\nStderr: 'RTNETLINK answers: >> >> File exists\\n'\n"] >> >> >> >> >> >> My guess is that this can be solved with the nova configuration. Sadly I >> >> am not too familiar with these. Any help would be great. >> >> Thanks >> >> Gary >> >> >> >> _______________________________________________ >> >> Mailing list: https://launchpad.net/~openstack >> >> Post to : openst...@lists.launchpad.net >> >> Unsubscribe : https://launchpad.net/~openstack >> >> More help : https://help.launchpad.net/ListHelp >> >> >> >> >> -- >> Mailing list: https://launchpad.net/~netstack >> Post to : netstack@lists.launchpad.net >> Unsubscribe : https://launchpad.net/~netstack >> More help : https://help.launchpad.net/ListHelp >> >> >> >> >> >> -- >> ~~~~~~~~~~~~~~~~~~~~~~~~~~~ >> Dan Wendlandt >> >> Nicira, Inc: www.nicira.com >> >> twitter: danwendlandt >> ~~~~~~~~~~~~~~~~~~~~~~~~~~~ >> >> >> > > > > -- > ~~~~~~~~~~~~~~~~~~~~~~~~~~~ > Dan Wendlandt > Nicira, Inc: www.nicira.com > twitter: danwendlandt > ~~~~~~~~~~~~~~~~~~~~~~~~~~~ > > > -- ~~~~~~~~~~~~~~~~~~~~~~~~~~~ Dan Wendlandt Nicira, Inc: www.nicira.com twitter: danwendlandt ~~~~~~~~~~~~~~~~~~~~~~~~~~~
-- Mailing list: https://launchpad.net/~netstack Post to : netstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~netstack More help : https://help.launchpad.net/ListHelp
