On Wed, May 16, 2012 at 2:01 PM, Sumit Naiksatam (snaiksat) < snaik...@cisco.com> wrote:
> The interesting data point here is that this only happens to the default > network created when the installation is done via DevStack. For all network > created subsequently I am not seeing this issue. I earlier thought that > this might have something to do with the exercises script getting executed > and possibly leaving a residue. However, in spite of not running the > exercises script I am still seeing this issue. It seems that for some > reason, the initialize_gateway() was getting called only when a new VM was > being created. But here it seems to be getting called before that (thus > leading to the creation of the bridge and the gateway devices), not sure > why. I am just trying to figure out how I can test the fix with respect to > the devstack setup (since this happens only on installation)… > Your explanation makes sense. I think there was a recent comment to automatically initialize all networks when nova-network is started, in case nova-network was started on a new host (e.g., in an HA configuration) or after a host reboot. Since devstack creates the network using nova-manage before starting nova-network, this would trigger the behavior your describe. For more info, see: https://github.com/openstack/nova/commit/30c8293ea7d4c9464e0139fb488e202fd0947f52 I suspect you should hit this every time you re-run stack.sh, no? > **** > > ** ** > > *From:* Dan Wendlandt [mailto:d...@nicira.com] > *Sent:* Wednesday, May 16, 2012 7:55 AM > *To:* gkot...@redhat.com > *Cc:* Sumit Naiksatam (snaiksat); netstack@lists.launchpad.net > *Subject:* Re: [Netstack] Fwd: [Openstack] [devstack/quantum] > Configuration issue**** > > ** ** > > ** ** > > On Wed, May 16, 2012 at 1:43 AM, Gary Kotton <gkot...@redhat.com> wrote:** > ** > > Thanks! > As far as I recall, and I may certain be wrong here, but in the past I did > not see the GW created until the first VM was deployed. Now I am seeing the > GW created when the various services are started. **** > > ** ** > > I have noticed this as well. I'm not aware of any changes to > QuantumManager that had this affect, though its possible someone pushed a > change that I didn't notice. I primarily use the OVS interface-driver and > didn't notice any issues, so I suspect Sumit is correct that we can at > least work around this with a change to the LB interface-driver (even > though the change that caused the issue was likely somewhere else). **** > > ** ** > > This is a good example of why we need Quantum integrated into devstack > commit-gating, as it will catch this type of complex integration issues. > Anyone have cycles to help push on this? I'm unlikely to make much > progress on it in the next week or two due to the F-1 release. **** > > ** ** > > Dan**** > > ** ** > > ** ** > > **** > > Thanks > Gary**** > > > > On 05/16/2012 10:54 AM, Sumit Naiksatam (snaiksat) wrote: **** > > This is probably not a devstack issue. The problem is with the LB gateway > driver wherein the IP address is trying to be set on a bridge device which > already has an IP address. The check for an existing bridge is being > performed, but the IP address is being set outside that check. Ideally, > this code should not have been invoked if the gateway was already set. > Something seems to have changed in the QuantumManager as result of which > this code is being invoked again. At any rate, I will fix the LB gateway > driver, and we will not see this.**** > > **** > > Thanks,**** > > ~Sumit.**** > > **** > > *From:* netstack-bounces+snaiksat=cisco....@lists.launchpad.net [ > mailto:netstack-bounces+snaiksat=cisco....@lists.launchpad.net<netstack-bounces+snaiksat=cisco....@lists.launchpad.net>] > *On Behalf Of *Gary Kotton > *Sent:* Monday, May 14, 2012 11:19 PM > *To:* <netstack@lists.launchpad.net> <netstack@lists.launchpad.net> > *Subject:* [Netstack] Fwd: [Openstack] [devstack/quantum] Configuration > issue**** > > **** > > > > -------- Original Message -------- **** > > *Subject: ***** > > [Openstack] [devstack/quantum] Configuration issue**** > > *Date: ***** > > Tue, 15 May 2012 09:00:52 +0300**** > > *From: ***** > > Gary Kotton <gkot...@redhat.com> <gkot...@redhat.com>**** > > *Reply-To: ***** > > gkot...@redhat.com**** > > *Organization: ***** > > Red Hat**** > > *To: ***** > > openst...@lists.launchpad.net**** > > **** > > Hi,**** > > This morning I encountered a problem (which did not happen a few days **** > > > ago :)). When devstack is launched, with quantum configured, the gateway **** > > and bridge devices are created. This causes problems with quantum.**** > > **** > > For example when devstack is up and running prior to deploying an **** > > instance we have:**** > > **** > > brq744ec2f4-c0 Link encap:Ethernet HWaddr fa:16:3e:03:a6:55**** > > inet addr:10.0.0.1 Bcast:0.0.0.0 Mask:255.255.255.0**** > > UP BROADCAST MULTICAST MTU:1500 Metric:1**** > > RX packets:0 errors:0 dropped:0 overruns:0 frame:0**** > > TX packets:0 errors:0 dropped:0 overruns:0 carrier:0**** > > collisions:0 txqueuelen:0**** > > RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)**** > > **** > > gw-744ec2f4-c0 Link encap:Ethernet HWaddr fa:16:3e:03:a6:55**** > > inet addr:10.0.0.1 Bcast:10.0.0.255 Mask:255.255.255.0**** > > UP BROADCAST MULTICAST MTU:1500 Metric:1**** > > RX packets:0 errors:0 dropped:0 overruns:0 frame:0**** > > TX packets:0 errors:0 dropped:0 overruns:0 carrier:0**** > > collisions:0 txqueuelen:500**** > > RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)**** > > **** > > When an instance is deployed the following happens:**** > > **** > > 2012-05-15 01:59:18 DEBUG nova.utils **** > > [req-4d50ed10-46e1-406c-9074-dc45da860365 **** > > df07eec326434b25800f3ebc17202fb3 2cafe0be4d7740098a89be39ffd1b72e] **** > > Running cmd (subprocess): sudo /usr/local/bin/nova-rootwrap ip address **** > > add 10.0.0.1/24 dev brq744ec2f4-c0 from (pid=4234) execute **** > > /opt/stack/nova/nova/utils.py:178**** > > 2012-05-15 01:59:18 DEBUG nova.utils **** > > [req-4d50ed10-46e1-406c-9074-dc45da860365 **** > > df07eec326434b25800f3ebc17202fb3 2cafe0be4d7740098a89be39ffd1b72e] **** > > Result was 254 from (pid=4234) execute /opt/stack/nova/nova/utils.py:194**** > > 2012-05-15 01:59:18 ERROR nova.rpc.amqp **** > > [req-4d50ed10-46e1-406c-9074-dc45da860365 **** > > df07eec326434b25800f3ebc17202fb3 2cafe0be4d7740098a89be39ffd1b72e] **** > > Exception during message handling**** > > 2012-05-15 01:59:18 TRACE nova.rpc.amqp Traceback (most recent call last):**** > > 2012-05-15 01:59:18 TRACE nova.rpc.amqp File **** > > "/opt/stack/nova/nova/rpc/amqp.py", line 263, in _process_data**** > > 2012-05-15 01:59:18 TRACE nova.rpc.amqp rval = **** > > node_func(context=ctxt, **node_args)**** > > 2012-05-15 01:59:18 TRACE nova.rpc.amqp File **** > > "/opt/stack/nova/nova/network/quantum/manager.py", line 390, in **** > > allocate_for_instance**** > > 2012-05-15 01:59:18 TRACE nova.rpc.amqp network, vif_rec, **** > > network['net_tenant_id'])**** > > 2012-05-15 01:59:18 TRACE nova.rpc.amqp File **** > > "/opt/stack/nova/nova/utils.py", line 880, in inner**** > > 2012-05-15 01:59:18 TRACE nova.rpc.amqp retval = f(*args, **kwargs)**** > > 2012-05-15 01:59:18 TRACE nova.rpc.amqp File **** > > "/opt/stack/nova/nova/network/quantum/manager.py", line 501, in > enable_dhcp**** > > 2012-05-15 01:59:18 TRACE nova.rpc.amqp **** > > self.l3driver.initialize_gateway(network_ref)**** > > 2012-05-15 01:59:18 TRACE nova.rpc.amqp File **** > > "/opt/stack/nova/nova/network/l3.py", line 98, in initialize_gateway**** > > 2012-05-15 01:59:18 TRACE nova.rpc.amqp **** > > gateway=(network_ref['gateway'] is not None))**** > > 2012-05-15 01:59:18 TRACE nova.rpc.amqp File **** > > "/opt/stack/nova/nova/network/linux_net.py", line 900, in plug**** > > 2012-05-15 01:59:18 TRACE nova.rpc.amqp return **** > > _get_interface_driver().plug(network, mac_address, gateway)**** > > 2012-05-15 01:59:18 TRACE nova.rpc.amqp File **** > > "/opt/stack/nova/nova/network/linux_net.py", line 1160, in plug**** > > 2012-05-15 01:59:18 TRACE nova.rpc.amqp run_as_root=True)**** > > 2012-05-15 01:59:18 TRACE nova.rpc.amqp File **** > > "/opt/stack/nova/nova/utils.py", line 201, in execute**** > > 2012-05-15 01:59:18 TRACE nova.rpc.amqp cmd=' '.join(cmd))**** > > 2012-05-15 01:59:18 TRACE nova.rpc.amqp ProcessExecutionError: **** > > Unexpected error while running command.**** > > 2012-05-15 01:59:18 TRACE nova.rpc.amqp Command: sudo **** > > /usr/local/bin/nova-rootwrap ip address add 10.0.0.1/24 dev brq744ec2f4-c0**** > > 2012-05-15 01:59:18 TRACE nova.rpc.amqp Exit code: 254**** > > 2012-05-15 01:59:18 TRACE nova.rpc.amqp Stdout: ''**** > > 2012-05-15 01:59:18 TRACE nova.rpc.amqp Stderr: 'RTNETLINK answers: File **** > > exists\n'**** > > 2012-05-15 01:59:18 TRACE nova.rpc.amqp**** > > 2012-05-15 01:59:18 ERROR nova.rpc.common **** > > [req-4d50ed10-46e1-406c-9074-dc45da860365 **** > > df07eec326434b25800f3ebc17202fb3 2cafe0be4d7740098a89be39ffd1b72e] **** > > Returning exception Unexpected error while running command.**** > > Command: sudo /usr/local/bin/nova-rootwrap ip address add 10.0.0.1/24 **** > > dev brq744ec2f4-c0**** > > Exit code: 254**** > > Stdout: ''**** > > Stderr: 'RTNETLINK answers: File exists\n' to caller**** > > 2012-05-15 01:59:18 ERROR nova.rpc.common **** > > [req-4d50ed10-46e1-406c-9074-dc45da860365 **** > > df07eec326434b25800f3ebc17202fb3 2cafe0be4d7740098a89be39ffd1b72e] **** > > ['Traceback (most recent call last):\n', ' File **** > > "/opt/stack/nova/nova/rpc/amqp.py", line 263, in _process_data\n rval **** > > = node_func(context=ctxt, **node_args)\n', ' File **** > > "/opt/stack/nova/nova/network/quantum/manager.py", line 390, in **** > > allocate_for_instance\n network, vif_rec, **** > > network[\'net_tenant_id\'])\n', ' File "/opt/stack/nova/nova/utils.py", **** > > line 880, in inner\n retval = f(*args, **kwargs)\n', ' File **** > > "/opt/stack/nova/nova/network/quantum/manager.py", line 501, in **** > > enable_dhcp\n self.l3driver.initialize_gateway(network_ref)\n', ' **** > > File "/opt/stack/nova/nova/network/l3.py", line 98, in **** > > initialize_gateway\n gateway=(network_ref[\'gateway\'] is not **** > > None))\n', ' File "/opt/stack/nova/nova/network/linux_net.py", line **** > > 900, in plug\n return _get_interface_driver().plug(network, **** > > mac_address, gateway)\n', ' File **** > > "/opt/stack/nova/nova/network/linux_net.py", line 1160, in plug\n **** > > run_as_root=True)\n', ' File "/opt/stack/nova/nova/utils.py", line 201, **** > > in execute\n cmd=\' \'.join(cmd))\n', "ProcessExecutionError: **** > > Unexpected error while running command.\nCommand: sudo **** > > /usr/local/bin/nova-rootwrap ip address add 10.0.0.1/24 dev **** > > brq744ec2f4-c0\nExit code: 254\nStdout: ''\nStderr: 'RTNETLINK answers: **** > > File exists\\n'\n"]**** > > **** > > **** > > My guess is that this can be solved with the nova configuration. Sadly I **** > > am not too familiar with these. Any help would be great.**** > > Thanks**** > > Gary**** > > **** > > _______________________________________________**** > > Mailing list: https://launchpad.net/~openstack**** > > Post to : openst...@lists.launchpad.net**** > > Unsubscribe : https://launchpad.net/~openstack**** > > More help : https://help.launchpad.net/ListHelp**** > > ** ** > > > -- > Mailing list: https://launchpad.net/~netstack > Post to : netstack@lists.launchpad.net > Unsubscribe : https://launchpad.net/~netstack > More help : https://help.launchpad.net/ListHelp**** > > > > **** > > ** ** > > -- > ~~~~~~~~~~~~~~~~~~~~~~~~~~~ > Dan Wendlandt **** > > Nicira, Inc: www.nicira.com**** > > twitter: danwendlandt > ~~~~~~~~~~~~~~~~~~~~~~~~~~~**** > > ** ** > -- ~~~~~~~~~~~~~~~~~~~~~~~~~~~ Dan Wendlandt Nicira, Inc: www.nicira.com twitter: danwendlandt ~~~~~~~~~~~~~~~~~~~~~~~~~~~
-- Mailing list: https://launchpad.net/~netstack Post to : netstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~netstack More help : https://help.launchpad.net/ListHelp
