Hello vpp-dev team, Because the root cause of the packet loss on virl1 and virl2 is not known up to now we decided to put virl1 out of production to not block vpp patches because of ssh timeouts during vpp-csit-verify tests.
It means that only virl3 server is in production so we can expect some delays in vpp-csit-verify jobs executions because of high load on the virl server. We will put the virl1 server back to the operation when root cause is known and issue fixed. Thank you very much for the understanding. Regards, Csit-dev team -----Original Message----- From: vpp-dev-boun...@lists.fd.io [mailto:vpp-dev-boun...@lists.fd.io] On Behalf Of Jan Gelety -X via RT Sent: Tuesday, June 20, 2017 17:20 Cc: csit-...@lists.fd.io; vpp-dev@lists.fd.io Subject: Re: [vpp-dev] [FD.io Helpdesk #41921] connection interruptiones between jenkins executor and VIRL servers Hello Anton, Thanks for the fast response. We will check local firewall setting as you proposed. Regards, Jan -----Original Message----- From: Anton Baranov via RT [mailto:fdio-helpd...@rt.linuxfoundation.org] Sent: Tuesday, June 20, 2017 17:13 To: Jan Gelety -X (jgelety - PANTHEON TECHNOLOGIES at Cisco) <jgel...@cisco.com> Cc: csit-...@lists.fd.io; vpp-dev@lists.fd.io Subject: [FD.io Helpdesk #41921] connection interruptiones between jenkins executor and VIRL servers Jan: This is what I got from fdio jenkins server (i did the tests with 10.30.{52,53}.2 hosts: $ ip ro get 10.30.52.2 10.30.52.2 via 10.30.48.1 dev eth0 src 10.30.48.5 cache The traffic is going directly through neutron router.. so we don't block any traffic on our firewall $ ping -q -c4 10.30.52.2 PING 10.30.52.2 (10.30.52.2) 56(84) bytes of data. --- 10.30.52.2 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3001ms rtt min/avg/max/mdev = 0.496/0.789/1.509/0.419 ms I was able to reach the host in 10.30.52.0/24 network from jenkins server $ nc -nv 10.30.52.2 22 Ncat: Version 6.40 ( http://nmap.org/ncat ) Ncat: Connection refused. Looks like access is blocked there. Could you check your local firewall setting and make sure you allow port 22/tcp ? The above is also true for 10.30.{53,54}.0/24 subnets Regards, On Tue Jun 20 10:51:15 2017, jgel...@cisco.com wrote: > Hello Vanessa, > > Thanks for the info. > > Just few remarks: > > 1. virl1 (10.30.51.28) - nodes of simulations started there are using > subnet 10.30.52.0/24 and we are experiencing ssh timeouts in this > subnet > > 2. virl2 (10.30.51.29) - nodes of simulations started there were > using subnet 10.30.53.0/24 and we were experiencing ssh timeouts in > this subnet; > - at the moment we switched the > subnet back to 10.30.51.0/24 and assigned there IP pool 10.30.51.106 - > 10.30.51.180 > - new tests started - let you > know the result tomorrow > > 3. virl3 (10.30.51.30) - nodes of simulations started there were using > subnet 10.30.51.0/24 and IP pool is set to 10.30.51.181 - 10.30.51.254 > and we didn't experience ssh timeouts in this subnet; > > > So would it be possible to check routes for subnets 10.30.52.0/24, > 10.30.53.0/24 and also for 10.30.54.0/24 (that is planned for vilr3 > when it will be upgraded)? > > Thank you very much. > > Regards, > Jan > > -----Original Message----- > From: Vanessa Valderrama via RT [mailto:fdio- > helpd...@rt.linuxfoundation.org] > Sent: Friday, June 16, 2017 22:01 > To: Jan Gelety -X (jgelety - PANTHEON TECHNOLOGIES at Cisco) > <jgel...@cisco.com> > Cc: csit-...@lists.fd.io; vpp-dev@lists.fd.io > Subject: [FD.io Helpdesk #41921] connection interruptiones between > jenkins executor and VIRL servers > > We did have the vendor run MTRs. I've attached the results. > > On Fri Jun 16 15:36:16 2017, valderrv wrote: > > Jan, > > > > I missed this conversation with abranov. Can this issue be resolved? > > > > <snip> > > <jgelety> abranov: unfortunatley I had no time to check test logs > > from last test cases before (because of meeting so I just had a look > > to console output) and I found out that ssh failures are not related > > to connection between jenkins and virl now (but it was this issue at > > the time I wrote the e-mail). > > <mnaser> [11:04:06] <jgelety> They are related to start up pf > > nested VM now - so I will ask VIRL support for the help here. > > </snip> > > > > Thank you, > > Vanessa > > > > On Fri Jun 16 12:12:43 2017, valderrv wrote: > > > Jan, > > > > > > We are looking into this issue. > > > > > > Thank you, > > > Vanessa > > > > > > On Fri Jun 16 09:12:55 2017, jgel...@cisco.com wrote: > > > > Hello Anton, > > > > > > > > Unfortunately we are still having issues with ssh connection > > > > timeouts during tests on virl. Could you, please, have a look on > > > > it? > > > > > > > > Thank you very much. > > > > Regards, > > > > Jan > > > > > > > > -----Original Message----- > > > > From: Anton Baranov via RT [mailto:fdio- > > > > helpd...@rt.linuxfoundation.org] > > > > Sent: Wednesday, June 14, 2017 15:45 > > > > To: Jan Gelety -X (jgelety - PANTHEON TECHNOLOGIES at Cisco) > > > > <jgel...@cisco.com> > > > > Cc: csit-...@lists.fd.io; vpp-dev@lists.fd.io > > > > Subject: [FD.io Helpdesk #41921] connection interruptiones > > > > between jenkins executor and VIRL servers > > > > > > > > Jan: > > > > > > > > On my side I currently don't see any connectivity problems > > > > between jenkins and VIRL servers. Please let me know if you're > > > > still having that issue. I'll keep an eye on that problem and if > > > > it reapears I'll report that to our cloud provider to check > > > > further. > > > > > > > > Thanks, > > > > -- > > > > Anton Baranov > > > > Systems and Network Administrator The Linux Foundation > > > > > > > > On Wed Jun 14 08:12:45 2017, jgel...@cisco.com wrote: > > > > > Dear held...@fd.io<mailto:held...@fd.io> > > > > > > > > > > We are observing connection issues between Jenkins executors > > > > > and VIRL servers that leads to failures of verify jobs > > > > > (https://jenkins.fd.io/view/vpp/job/vpp-csit-verify-virl- > > > > > master/ > > > > > , > > > > > https://jenkins.fd.io/view/csit/job/csit-vpp-functional-master > > > > > - ubuntu1604-virl/, > > > > > https://jenkins.fd.io/view/csit/job/csit-vpp- > > > > > functional-master-centos7-virl/) because of ssh connection > > > > > timeouts. > > > > > > > > > > Could you, please, have a look on it? > > > > > > > > > > Thank you very much. > > > > > > > > > > Regards, > > > > > Jan > > > > > > > > > > > > > > > > > > > > > -- Anton Baranov Systems and Network Administrator The Linux Foundation _______________________________________________ vpp-dev mailing list vpp-dev@lists.fd.io https://lists.fd.io/mailman/listinfo/vpp-dev _______________________________________________ vpp-dev mailing list vpp-dev@lists.fd.io https://lists.fd.io/mailman/listinfo/vpp-dev
