On Tue, Apr 1, 2014 at 4:34 PM, Florin Dumitrascu < florin.dumitra...@intunenetworks.com> wrote:
> So I have this failure now which seems to be consistent. > A few details about the setup: > > Host 5 is standalone (single host in cluster). Host 15 is in a cluster > with host 16. There is 1 VM instance running on host 5, 1 VM running on > host 16, none running on host 15. > The tunnels between hosts 5 and 16 have been created (on second attempt, > first time, I got the error in previous email). > Now I am starting a new instance on host 15 and the tunnel creation fails. > I know there was an issue with the tunnels on second host in the cluster > (VM_REQUIRES_NETWORK), has this not been fixed ? > Do you think the error in the ovstunnel.log on host 5 below is related ? > > There is still issue with tunnels between the hosts in the same cluster. There was not enough time when bug was reported and RC was already cut out for 4.3. But that was XAPI issue where network object exists but corresponding bridge is not created. From the below logs, clearly bridge is created problem seems to be with OVS. '-1' is returned as OF port for the tunnel interface. I am running some tests, i will run this scenario see if I can repro it. Can you please run below commands on host 5 and share the results /usr/bin/ovs-vsctl get interface f4344f02-be53-4297-9344-5d799c41c8bf ofport /usr/bin/ovs-vsctl get interface t213-5-15 ofport > --------- > 2014-04-01 11:16:07 DEBUG [root] #### VMOPS enter create_tunnel #### > 2014-04-01 11:16:07 DEBUG [root] Entering create_tunnel > 2014-04-01 11:16:07 DEBUG [root] Executing:['/usr/bin/ovs-vsctl', > '--timeout=30', 'wait-until', 'bridge', 'xapi5', '--', 'get', 'bridge', > 'xapi5', 'name'] > 2014-04-01 11:16:07 DEBUG [root] bridge xapi5 for creating tunnel - > VERIFIED > 2014-04-01 11:16:07 DEBUG [root] Executing:['/usr/bin/ovs-vsctl', > 'add-port', 'xapi5', 't213-5-15', '--', 'set', 'interface', 't213-5-15', > 'type=gre', 'options:key=213', 'options:remote_ip=10.20.1.22'] > 2014-04-01 11:16:07 DEBUG [root] Executing:['/usr/bin/ovs-vsctl', > 'get', 'port', 't213-5-15', 'interfaces'] > 2014-04-01 11:16:07 DEBUG [root] Executing:['/usr/bin/ovs-vsctl', > 'get', 'interface', 'f4344f02-be53-4297-9344-5d799c41c8bf', 'options:key'] > 2014-04-01 11:16:07 DEBUG [root] Executing:['/usr/bin/ovs-vsctl', > 'get', 'interface', 'f4344f02-be53-4297-9344-5d799c41c8bf', > 'options:remote_ip'] > 2014-04-01 11:16:07 DEBUG [root] Tunnel interface > validated:['/usr/bin/ovs-vsctl', 'get', 'interface', > 'f4344f02-be53-4297-9344-5d799c41c8bf', 'options:remote_ip'] > 2014-04-01 11:16:07 DEBUG [root] Executing:['/usr/bin/ovs-vsctl', > 'get', 'interface', 'f4344f02-be53-4297-9344-5d799c41c8bf', 'ofport'] > 2014-04-01 11:16:07 DEBUG [root] Executing:['/usr/bin/ovs-ofctl', > 'add-flow', 'xapi5', > 'hard_timeout=0,idle_timeout=0,priority=1000,in_port=-1,dl_dst=ff:ff:ff:ff:ff:ff,actions=drop'] > 2014-04-01 11:16:07 DEBUG [root] The command exited with the error > code: 1 (stderr output:ovs-ofctl: -1: negative values not supported for > in_port > ) > 2014-04-01 11:16:07 DEBUG [root] An unexpected error occured. Rolling > back > 2014-04-01 11:16:07 DEBUG [root] Deleting GRE interface > 2014-04-01 11:16:07 DEBUG [root] Executing:['/usr/bin/ovs-vsctl', > 'del-port', 'xapi5', 't213-5-15'] > ---------- > > Thanks, > Florin > > -----Original Message----- > From: Florin Dumitrascu [mailto:florin.dumitra...@intunenetworks.com] > Sent: Tuesday, April 01, 2014 9:37 AM > To: dev@cloudstack.apache.org > Subject: RE: OVS plugin communication failure (CS 4.3.0) > > Murali, Benoit, > > First, thanks for the tip, I knew about /var/log/ovstunnel.log from CS > 4.1.1 but had no idea it moved under "cloud" folder :) I am on a fresh > installed setup. > > So, looking at this log, I am seeing something that looks like this: > > ---- > 2014-03-31 22:01:26 DEBUG [root] Entering create_tunnel > 2014-03-31 22:01:26 DEBUG [root] Executing:['/usr/bin/ovs-vsctl', > '--timeout=30', 'wait-until', 'bridge', 'xapi4', '--', 'get', 'bridge', > 'xapi4', 'name'] > 2014-03-31 22:01:56 DEBUG [root] The command exited with the error > code: -14 (stderr output:) > ---- > > This is not happening all the time. A second attempt to create an instance > did create the tunnel successfully. > I was actually seeing something similar in our previous setup which was CS > 4.1.1 + XenServer 6.1.0 > > > Florin > > -----Original Message----- > From: benoit lair [mailto:kurushi4...@gmail.com] > Sent: Tuesday, April 01, 2014 9:22 AM > To: dev@cloudstack.apache.org > Subject: Re: OVS plugin communication failure (CS 4.3.0) > > Hello Florin, > > > Are you on a fresh install or on an upgraded version (from 4.1 ?) > > > Regards, Benoit. > > > 2014-04-01 9:58 GMT+02:00 Murali Reddy <murali.re...@citrix.com>: > > > On 01/04/14 3:46 AM, "Florin Dumitrascu" > > <florin.dumitra...@intunenetworks.com> wrote: > > > > >Hi, > > > > > >I am using CS 4.3.0 and XenServer 6.2.0 with SP1, on a GRE isolated > > >network. > > >The tunnel creation keeps failing intermittently with the error below > > >(failure communicating with the plugin). > > >Cloudstack management server can ping the xenserver reliably, there > > >doesn't seem to be any issue at the IP level. > > >Any idea what is causing this ? > > > > Do you see any relevant errors in /var/log/cloud/ovstunnel.log or > > /var/log/xensource.log > > > > > > > >Thanks > > > > > >2014-03-31 23:08:56,528 WARN [c.c.h.x.r.CitrixResourceBase] > > >(DirectAgent-265:ctx-3ca23292) Caught execption when creating ovs > > >tunnel > > >com.cloud.utils.exception.CloudRuntimeException: callHostPlugin > > >failed for cmd: create_tunnel with args to: 5, remote_ip: 10.20.1.20, > > >from: 16, bridge: xapi4, key: 213, due to There was a failure > > >communicating with the plugin. > > > at > > >com.cloud.hypervisor.xen.resource.CitrixResourceBase.callHostPlugin(C > > >itrix > > >ResourceBase.java:4239) > > > at > > >com.cloud.hypervisor.xen.resource.CitrixResourceBase.execute(CitrixRe > > >sourc > > >eBase.java:5685) > > > at > > >com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(C > > >itrix > > >ResourceBase.java:567) > > > at > > >com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest( > > >XenSe > > >rver56Resource.java:59) > > > at > > >com.cloud.hypervisor.xen.resource.XenServer610Resource.executeRequest > > >(XenS > > >erver610Resource.java:106) > > > at > > >com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAg > > >entAt > > >tache.java:216) > > > > > >Florin > > >IMPORTANT NOTE: The information in this e-mail (and any attachments) > > >is confidential. The contents may not be disclosed or used by anyone > > >other than the addressee. If you are not the intended recipient, > > >please notify the sender immediately or telephone: +353 (0)1 6204700. > > >We cannot accept any responsibility for the accuracy or completeness > > >of this message as it has been transmitted over a public network. If > > >you suspect that the message may have been intercepted or amended, > please call the sender. > > > > > > > > > > > IMPORTANT NOTE: The information in this e-mail (and any attachments) is > confidential. The contents may not be disclosed or used by anyone other > than the addressee. If you are not the intended recipient, please notify > the sender immediately or telephone: +353 (0)1 6204700. We cannot accept > any responsibility for the accuracy or completeness of this message as it > has been transmitted over a public network. If you suspect that the message > may have been intercepted or amended, please call the sender. > > IMPORTANT NOTE: The information in this e-mail (and any attachments) is > confidential. The contents may not be disclosed or used by anyone other > than the addressee. If you are not the intended recipient, please notify > the sender immediately or telephone: +353 (0)1 6204700. We cannot accept > any responsibility for the accuracy or completeness of this message as it > has been transmitted over a public network. If you suspect that the message > may have been intercepted or amended, please call the sender. >
