On Thu, May 17, 2012 at 10:24 AM, Dan Wendlandt <d...@nicira.com> wrote:
> Hi Gary, > > Thanks for reporting this. > > After some digging, the bug seems to have been introduced by an error in a > blanket changeset to switch Nova from using json to jasonutils. During > that translation, a "return" statement was dropped, meaning the serializer > always returned Null regardless of what data was pushed in. See: > https://github.com/openstack/nova/commit/ba76b954e69de56f76f9db5cade1780bc351be67#L5L42 > > Another bug where having Quantum be part of the gating process would have > helped... so we really need to find resources to help get that work > complete. > > Patch here is trivial, so I'll push it and hopefully it can get reviewed > quickly by the Nova core team. > review link: https://review.openstack.org/#/c/7550/ > > Dan > > > > > On Thu, May 17, 2012 at 12:24 AM, Gary Kotton <gkot...@redhat.com> wrote: > >> ** >> Hi, >> Running devstack this morning the following happens: >> >> + /opt/stack/nova/bin/nova-manage network create private 10.0.0.0/24 1 >> 256 >> 2012-05-17 03:22:05 WARNING nova.network.quantum.manager >> [req-5458d926-54d1-4877-9c46-d04035742eaf None None] QuantumManager does >> not use 'bridge' parameter. >> 2012-05-17 03:22:05 WARNING nova.network.quantum.manager >> [req-5458d926-54d1-4877-9c46-d04035742eaf None None] QuantumManager does >> not use 'bridge_interface' parameter. >> 2012-05-17 03:22:05 DEBUG nova.network.quantum.quantum_connection >> [req-5458d926-54d1-4877-9c46-d04035742eaf None None] Quantum Client >> Request: POST /v1.1/tenants/default/networks.json from (pid=3826) >> do_request /opt/stack/nova/nova/network/quantum/client.py:178 >> 2012-05-17 03:22:05 DEBUG nova.network.quantum.quantum_connection >> [req-5458d926-54d1-4877-9c46-d04035742eaf None None] Quantum Client Reply >> (code = 400) : >> {"Error": {"message": "The server could not comply with the request >> since it is either malformed or otherwise incorrect.", "code": 400}} from >> (pid=3826) do_request /opt/stack/nova/nova/network/quantum/client.py:189 >> Command failed, please check log for more info >> 2012-05-17 03:22:05 CRITICAL nova >> [req-5458d926-54d1-4877-9c46-d04035742eaf None None] Server 400 error: >> {"Error": {"message": "The server could not comply with the request since >> it is either malformed or otherwise incorrect.", "code": 400}} >> 2012-05-17 03:22:05 TRACE nova Traceback (most recent call last): >> 2012-05-17 03:22:05 TRACE nova File "/opt/stack/nova/bin/nova-manage", >> line 1755, in <module> >> 2012-05-17 03:22:05 TRACE nova main() >> 2012-05-17 03:22:05 TRACE nova File "/opt/stack/nova/bin/nova-manage", >> line 1742, in main >> 2012-05-17 03:22:05 TRACE nova fn(*fn_args, **fn_kwargs) >> 2012-05-17 03:22:05 TRACE nova File "/opt/stack/nova/bin/nova-manage", >> line 817, in create >> 2012-05-17 03:22:05 TRACE nova fixed_cidr=fixed_cidr) >> 2012-05-17 03:22:05 TRACE nova File >> "/opt/stack/nova/nova/network/quantum/manager.py", line 232, in >> create_networks >> 2012-05-17 03:22:05 TRACE nova nova_id=nova_id) >> 2012-05-17 03:22:05 TRACE nova File >> "/opt/stack/nova/nova/network/quantum/quantum_connection.py", line 67, in >> create_network >> 2012-05-17 03:22:05 TRACE nova resdict = >> self.client.create_network(data, tenant=tenant_id) >> 2012-05-17 03:22:05 TRACE nova File >> "/opt/stack/nova/nova/network/quantum/client.py", line 82, in with_params >> 2012-05-17 03:22:05 TRACE nova ret = self.func(instance, *args, >> **kwargs) >> 2012-05-17 03:22:05 TRACE nova File >> "/opt/stack/nova/nova/network/quantum/client.py", line 250, in >> create_network >> 2012-05-17 03:22:05 TRACE nova return self.do_request("POST", >> self.networks_path, body=body) >> 2012-05-17 03:22:05 TRACE nova File >> "/opt/stack/nova/nova/network/quantum/client.py", line 204, in do_request >> 2012-05-17 03:22:05 TRACE nova % locals()) >> 2012-05-17 03:22:05 TRACE nova QuantumServerException: Server 400 error: >> {"Error": {"message": "The server could not comply with the request since >> it is either malformed or otherwise incorrect.", "code": 400}} >> 2012-05-17 03:22:05 TRACE nova >> ++ failed >> ++ local r=1 >> ++ set +o xtrace >> >> Thanks >> Gary >> >> >> On 05/17/2012 12:07 AM, Dan Wendlandt wrote: >> >> >> >> On Wed, May 16, 2012 at 2:01 PM, Sumit Naiksatam (snaiksat) < >> snaik...@cisco.com> wrote: >> >>> The interesting data point here is that this only happens to the >>> default network created when the installation is done via DevStack. For all >>> network created subsequently I am not seeing this issue. I earlier thought >>> that this might have something to do with the exercises script getting >>> executed and possibly leaving a residue. However, in spite of not running >>> the exercises script I am still seeing this issue. It seems that for some >>> reason, the initialize_gateway() was getting called only when a new VM was >>> being created. But here it seems to be getting called before that (thus >>> leading to the creation of the bridge and the gateway devices), not sure >>> why. I am just trying to figure out how I can test the fix with respect to >>> the devstack setup (since this happens only on installation)… >>> >> >> Your explanation makes sense. I think there was a recent comment to >> automatically initialize all networks when nova-network is started, in case >> nova-network was started on a new host (e.g., in an HA configuration) or >> after a host reboot. Since devstack creates the network using nova-manage >> before starting nova-network, this would trigger the behavior your >> describe. For more info, see: >> https://github.com/openstack/nova/commit/30c8293ea7d4c9464e0139fb488e202fd0947f52 >> >> I suspect you should hit this every time you re-run stack.sh, no? >> >> >> >> >> >>> >>> >>> *From:* Dan Wendlandt [mailto:d...@nicira.com] >>> *Sent:* Wednesday, May 16, 2012 7:55 AM >>> *To:* gkot...@redhat.com >>> *Cc:* Sumit Naiksatam (snaiksat); netstack@lists.launchpad.net >>> *Subject:* Re: [Netstack] Fwd: [Openstack] [devstack/quantum] >>> Configuration issue >>> >>> >>> >>> >>> >>> On Wed, May 16, 2012 at 1:43 AM, Gary Kotton <gkot...@redhat.com> wrote: >>> >>> Thanks! >>> As far as I recall, and I may certain be wrong here, but in the past I >>> did not see the GW created until the first VM was deployed. Now I am seeing >>> the GW created when the various services are started. >>> >>> >>> >>> I have noticed this as well. I'm not aware of any changes to >>> QuantumManager that had this affect, though its possible someone pushed a >>> change that I didn't notice. I primarily use the OVS interface-driver and >>> didn't notice any issues, so I suspect Sumit is correct that we can at >>> least work around this with a change to the LB interface-driver (even >>> though the change that caused the issue was likely somewhere else). >>> >>> >>> >>> This is a good example of why we need Quantum integrated into devstack >>> commit-gating, as it will catch this type of complex integration issues. >>> Anyone have cycles to help push on this? I'm unlikely to make much >>> progress on it in the next week or two due to the F-1 release. >>> >>> >>> >>> Dan >>> >>> >>> >>> >>> >>> >>> >>> Thanks >>> Gary >>> >>> >>> >>> On 05/16/2012 10:54 AM, Sumit Naiksatam (snaiksat) wrote: >>> >>> This is probably not a devstack issue. The problem is with the LB >>> gateway driver wherein the IP address is trying to be set on a bridge >>> device which already has an IP address. The check for an existing bridge is >>> being performed, but the IP address is being set outside that check. >>> Ideally, this code should not have been invoked if the gateway was already >>> set. Something seems to have changed in the QuantumManager as result of >>> which this code is being invoked again. At any rate, I will fix the LB >>> gateway driver, and we will not see this. >>> >>> >>> >>> Thanks, >>> >>> ~Sumit. >>> >>> >>> >>> *From:* netstack-bounces+snaiksat=cisco....@lists.launchpad.net [ >>> mailto:netstack-bounces+snaiksat=cisco....@lists.launchpad.net<netstack-bounces+snaiksat=cisco....@lists.launchpad.net>] >>> *On Behalf Of *Gary Kotton >>> *Sent:* Monday, May 14, 2012 11:19 PM >>> *To:* <netstack@lists.launchpad.net> <netstack@lists.launchpad.net> >>> *Subject:* [Netstack] Fwd: [Openstack] [devstack/quantum] Configuration >>> issue >>> >>> >>> >>> >>> >>> -------- Original Message -------- >>> >>> *Subject: * >>> >>> [Openstack] [devstack/quantum] Configuration issue >>> >>> *Date: * >>> >>> Tue, 15 May 2012 09:00:52 +0300 >>> >>> *From: * >>> >>> Gary Kotton <gkot...@redhat.com> <gkot...@redhat.com> >>> >>> *Reply-To: * >>> >>> gkot...@redhat.com >>> >>> *Organization: * >>> >>> Red Hat >>> >>> *To: * >>> >>> openst...@lists.launchpad.net >>> >>> >>> >>> Hi, >>> >>> This morning I encountered a problem (which did not happen a few days >>> >>> ago :)). When devstack is launched, with quantum configured, the gateway >>> >>> and bridge devices are created. This causes problems with quantum. >>> >>> >>> >>> For example when devstack is up and running prior to deploying an >>> >>> instance we have: >>> >>> >>> >>> brq744ec2f4-c0 Link encap:Ethernet HWaddr fa:16:3e:03:a6:55 >>> >>> inet addr:10.0.0.1 Bcast:0.0.0.0 Mask:255.255.255.0 >>> >>> UP BROADCAST MULTICAST MTU:1500 Metric:1 >>> >>> RX packets:0 errors:0 dropped:0 overruns:0 frame:0 >>> >>> TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 >>> >>> collisions:0 txqueuelen:0 >>> >>> RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) >>> >>> >>> >>> gw-744ec2f4-c0 Link encap:Ethernet HWaddr fa:16:3e:03:a6:55 >>> >>> inet addr:10.0.0.1 Bcast:10.0.0.255 Mask:255.255.255.0 >>> >>> UP BROADCAST MULTICAST MTU:1500 Metric:1 >>> >>> RX packets:0 errors:0 dropped:0 overruns:0 frame:0 >>> >>> TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 >>> >>> collisions:0 txqueuelen:500 >>> >>> RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) >>> >>> >>> >>> When an instance is deployed the following happens: >>> >>> >>> >>> 2012-05-15 01:59:18 DEBUG nova.utils >>> >>> [req-4d50ed10-46e1-406c-9074-dc45da860365 >>> >>> df07eec326434b25800f3ebc17202fb3 2cafe0be4d7740098a89be39ffd1b72e] >>> >>> Running cmd (subprocess): sudo /usr/local/bin/nova-rootwrap ip address >>> >>> add 10.0.0.1/24 dev brq744ec2f4-c0 from (pid=4234) execute >>> >>> /opt/stack/nova/nova/utils.py:178 >>> >>> 2012-05-15 01:59:18 DEBUG nova.utils >>> >>> [req-4d50ed10-46e1-406c-9074-dc45da860365 >>> >>> df07eec326434b25800f3ebc17202fb3 2cafe0be4d7740098a89be39ffd1b72e] >>> >>> Result was 254 from (pid=4234) execute /opt/stack/nova/nova/utils.py:194 >>> >>> 2012-05-15 01:59:18 ERROR nova.rpc.amqp >>> >>> [req-4d50ed10-46e1-406c-9074-dc45da860365 >>> >>> df07eec326434b25800f3ebc17202fb3 2cafe0be4d7740098a89be39ffd1b72e] >>> >>> Exception during message handling >>> >>> 2012-05-15 01:59:18 TRACE nova.rpc.amqp Traceback (most recent call last): >>> >>> 2012-05-15 01:59:18 TRACE nova.rpc.amqp File >>> >>> "/opt/stack/nova/nova/rpc/amqp.py", line 263, in _process_data >>> >>> 2012-05-15 01:59:18 TRACE nova.rpc.amqp rval = >>> >>> node_func(context=ctxt, **node_args) >>> >>> 2012-05-15 01:59:18 TRACE nova.rpc.amqp File >>> >>> "/opt/stack/nova/nova/network/quantum/manager.py", line 390, in >>> >>> allocate_for_instance >>> >>> 2012-05-15 01:59:18 TRACE nova.rpc.amqp network, vif_rec, >>> >>> network['net_tenant_id']) >>> >>> 2012-05-15 01:59:18 TRACE nova.rpc.amqp File >>> >>> "/opt/stack/nova/nova/utils.py", line 880, in inner >>> >>> 2012-05-15 01:59:18 TRACE nova.rpc.amqp retval = f(*args, **kwargs) >>> >>> 2012-05-15 01:59:18 TRACE nova.rpc.amqp File >>> >>> "/opt/stack/nova/nova/network/quantum/manager.py", line 501, in enable_dhcp >>> >>> 2012-05-15 01:59:18 TRACE nova.rpc.amqp >>> >>> self.l3driver.initialize_gateway(network_ref) >>> >>> 2012-05-15 01:59:18 TRACE nova.rpc.amqp File >>> >>> "/opt/stack/nova/nova/network/l3.py", line 98, in initialize_gateway >>> >>> 2012-05-15 01:59:18 TRACE nova.rpc.amqp >>> >>> gateway=(network_ref['gateway'] is not None)) >>> >>> 2012-05-15 01:59:18 TRACE nova.rpc.amqp File >>> >>> "/opt/stack/nova/nova/network/linux_net.py", line 900, in plug >>> >>> 2012-05-15 01:59:18 TRACE nova.rpc.amqp return >>> >>> _get_interface_driver().plug(network, mac_address, gateway) >>> >>> 2012-05-15 01:59:18 TRACE nova.rpc.amqp File >>> >>> "/opt/stack/nova/nova/network/linux_net.py", line 1160, in plug >>> >>> 2012-05-15 01:59:18 TRACE nova.rpc.amqp run_as_root=True) >>> >>> 2012-05-15 01:59:18 TRACE nova.rpc.amqp File >>> >>> "/opt/stack/nova/nova/utils.py", line 201, in execute >>> >>> 2012-05-15 01:59:18 TRACE nova.rpc.amqp cmd=' '.join(cmd)) >>> >>> 2012-05-15 01:59:18 TRACE nova.rpc.amqp ProcessExecutionError: >>> >>> Unexpected error while running command. >>> >>> 2012-05-15 01:59:18 TRACE nova.rpc.amqp Command: sudo >>> >>> /usr/local/bin/nova-rootwrap ip address add 10.0.0.1/24 dev brq744ec2f4-c0 >>> >>> 2012-05-15 01:59:18 TRACE nova.rpc.amqp Exit code: 254 >>> >>> 2012-05-15 01:59:18 TRACE nova.rpc.amqp Stdout: '' >>> >>> 2012-05-15 01:59:18 TRACE nova.rpc.amqp Stderr: 'RTNETLINK answers: File >>> >>> exists\n' >>> >>> 2012-05-15 01:59:18 TRACE nova.rpc.amqp >>> >>> 2012-05-15 01:59:18 ERROR nova.rpc.common >>> >>> [req-4d50ed10-46e1-406c-9074-dc45da860365 >>> >>> df07eec326434b25800f3ebc17202fb3 2cafe0be4d7740098a89be39ffd1b72e] >>> >>> Returning exception Unexpected error while running command. >>> >>> Command: sudo /usr/local/bin/nova-rootwrap ip address add 10.0.0.1/24 >>> >>> dev brq744ec2f4-c0 >>> >>> Exit code: 254 >>> >>> Stdout: '' >>> >>> Stderr: 'RTNETLINK answers: File exists\n' to caller >>> >>> 2012-05-15 01:59:18 ERROR nova.rpc.common >>> >>> [req-4d50ed10-46e1-406c-9074-dc45da860365 >>> >>> df07eec326434b25800f3ebc17202fb3 2cafe0be4d7740098a89be39ffd1b72e] >>> >>> ['Traceback (most recent call last):\n', ' File >>> >>> "/opt/stack/nova/nova/rpc/amqp.py", line 263, in _process_data\n rval >>> >>> = node_func(context=ctxt, **node_args)\n', ' File >>> >>> "/opt/stack/nova/nova/network/quantum/manager.py", line 390, in >>> >>> allocate_for_instance\n network, vif_rec, >>> >>> network[\'net_tenant_id\'])\n', ' File "/opt/stack/nova/nova/utils.py", >>> >>> line 880, in inner\n retval = f(*args, **kwargs)\n', ' File >>> >>> "/opt/stack/nova/nova/network/quantum/manager.py", line 501, in >>> >>> enable_dhcp\n self.l3driver.initialize_gateway(network_ref)\n', ' >>> >>> File "/opt/stack/nova/nova/network/l3.py", line 98, in >>> >>> initialize_gateway\n gateway=(network_ref[\'gateway\'] is not >>> >>> None))\n', ' File "/opt/stack/nova/nova/network/linux_net.py", line >>> >>> 900, in plug\n return _get_interface_driver().plug(network, >>> >>> mac_address, gateway)\n', ' File >>> >>> "/opt/stack/nova/nova/network/linux_net.py", line 1160, in plug\n >>> >>> run_as_root=True)\n', ' File "/opt/stack/nova/nova/utils.py", line 201, >>> >>> in execute\n cmd=\' \'.join(cmd))\n', "ProcessExecutionError: >>> >>> Unexpected error while running command.\nCommand: sudo >>> >>> /usr/local/bin/nova-rootwrap ip address add 10.0.0.1/24 dev >>> >>> brq744ec2f4-c0\nExit code: 254\nStdout: ''\nStderr: 'RTNETLINK answers: >>> >>> File exists\\n'\n"] >>> >>> >>> >>> >>> >>> My guess is that this can be solved with the nova configuration. Sadly I >>> >>> am not too familiar with these. Any help would be great. >>> >>> Thanks >>> >>> Gary >>> >>> >>> >>> _______________________________________________ >>> >>> Mailing list: https://launchpad.net/~openstack >>> >>> Post to : openst...@lists.launchpad.net >>> >>> Unsubscribe : https://launchpad.net/~openstack >>> >>> More help : https://help.launchpad.net/ListHelp >>> >>> >>> >>> >>> -- >>> Mailing list: https://launchpad.net/~netstack >>> Post to : netstack@lists.launchpad.net >>> Unsubscribe : https://launchpad.net/~netstack >>> More help : https://help.launchpad.net/ListHelp >>> >>> >>> >>> >>> >>> -- >>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~ >>> Dan Wendlandt >>> >>> Nicira, Inc: www.nicira.com >>> >>> twitter: danwendlandt >>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~ >>> >>> >>> >> >> >> >> -- >> ~~~~~~~~~~~~~~~~~~~~~~~~~~~ >> Dan Wendlandt >> Nicira, Inc: www.nicira.com >> twitter: danwendlandt >> ~~~~~~~~~~~~~~~~~~~~~~~~~~~ >> >> >> > > > -- > ~~~~~~~~~~~~~~~~~~~~~~~~~~~ > Dan Wendlandt > Nicira, Inc: www.nicira.com > twitter: danwendlandt > ~~~~~~~~~~~~~~~~~~~~~~~~~~~ > > -- ~~~~~~~~~~~~~~~~~~~~~~~~~~~ Dan Wendlandt Nicira, Inc: www.nicira.com twitter: danwendlandt ~~~~~~~~~~~~~~~~~~~~~~~~~~~
-- Mailing list: https://launchpad.net/~netstack Post to : netstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~netstack More help : https://help.launchpad.net/ListHelp
