Hey maciek, We don’t need and would prefer NOT to remove that .1 addresses on those virtual routers.
Just the addition of the static is whats needed right now.. thanks, Ed > On Jun 28, 2017, at 5:07 AM, Maciek Konstantynowicz (mkonstan) > <mkons...@cisco.com> wrote: > > Anton and Team, > > The continued interruptions of IP connectivity to/from VIRL server > simulations on the management subnet have been impacting both CSIT and VPP > project operations. We decided to temporarily remove VPP VIRL based verify > jobs, job/vpp-csit-verify-virl-master/, from both per vpp patch auto-trigger > and the voting rights - Ed W. was kind to prepared required ci-mgmt patches, > but they are not merged yet (https://gerrit.fd.io/r/#/c/7319/, > https://gerrit.fd.io/r/#/c/7320/). > > Before we proceed with above step, we want to do one more set of network > infra focused tests per yesterday exchange on #fdio-infra irc with > Vanessa/valderrv, Ed Kern/snergster and Mohammed/mnaser. Here quick recap: > > Connectivity is affected between following the mgmt subnets added few weeks > back as part of [FD.io Helpdesk #40733]: > 10.30.52.0/24 > 10.30.53.0/24 > 10.30.54.0/24 > > The high packet drop rate (50..70%) problem seem to occur sporadically, but > if packets are passing thru the default gateway router that has address .1 in > each of above subnets. This affects all connectivity to jenkins slaves, but > also between tb4 virl hosts. The problem is never observed if packets are > sent directly between the hosts, it works fine. > > Test proposal: > > Configure the router that acts as default gateway for these subnet with the > following static routes: > 10.30.52.0/24 at 10.30.51.28 // tb-4virl1 mgmt addr > 10.30.53.0/24 at 10.30.51.29 // tb-4virl1 mgmt addr > 10.30.54.0/24 at 10.30.51.30 // tb-4virl1 mgmt addr > Meaning all packets to above subnets will be routed through the main > management IP address on respective tb4-virl host, per wiki [1]. > This will remove default gateway router from the problem domain under > investigation. > this is all well and good > Remove following IP addresses > from the default gateway router: > 10.30.52.1/24 > 10.30.53.1/24 > 10.30.54.1/24 > Not sure how this got in there….we want to keep these right where they are unless there is some reason to remove them > Continue to advertise below routes into WAN to ensure reachability from > Jenkins slave and LF FD.io infra: > 10.30.52.0/24 > 10.30.53.0/24 > 10.30.54.0/24 > this is also correct... > Could you pls advise when can these be conducted? > > -Maciek > > [1] > https://wiki.fd.io/view/CSIT/CSIT_LF_testbed#Management_VLAN_IP_Addresses_allocation > >> On 21 Jun 2017, at 16:00, Jan Gelety -X (jgelety - PANTHEON TECHNOLOGIES at >> Cisco) <jgel...@cisco.com> wrote: >> >> Hello Anton, >> >> We did some checks and here are results: >> >> 1. ping simulated node from the host itself - ping is OK >> >> 2. ping simulated node from other host (i.e. node simulated on virl2, >> executing ping command on virl3) - discovered packet loss (see e-mail from >> Peter below) >> - even for successful ping packet transition we can see the wide range of >> time - from cca 0,6ms to 45ms... >> >> We are still investigating VIRL settings but do you have some hints for us? >> >> Thanks, >> Jan >> >> -----Original Message----- >> From: Peter Mikus -X (pmikus - PANTHEON TECHNOLOGIES at Cisco) >> Sent: Wednesday, June 21, 2017 15:20 >> To: Jan Gelety -X (jgelety - PANTHEON TECHNOLOGIES at Cisco) >> <jgel...@cisco.com> >> Subject: RE: [vpp-dev] [FD.io Helpdesk #41921] connection interruptiones >> between jenkins executor and VIRL servers >> >> virl@t4-virl3:/home/testuser$ ping 10.30.51.127 >> PING 10.30.51.127 (10.30.51.127) 56(84) bytes of data. >> 64 bytes from 10.30.51.127: icmp_seq=54 ttl=64 time=1.86 ms >> ... >> ^C >> --- 10.30.51.127 ping statistics --- >> 1202 packets transmitted, 193 received, 83% packet loss, time 1202345ms >> rtt min/avg/max/mdev = 0.369/0.736/3.271/0.509 ms >> virl@t4-virl3:/home/testuser$ ping 10.30.51.29 >> >> Peter Mikus >> Engineer - Software >> Cisco Systems Limited >> >> -----Original Message----- >> From: vpp-dev-boun...@lists.fd.io [mailto:vpp-dev-boun...@lists.fd.io] On >> Behalf Of Jan Gelety -X via RT >> Sent: Tuesday, June 20, 2017 5:20 PM >> Cc: csit-...@lists.fd.io; vpp-dev@lists.fd.io >> Subject: Re: [vpp-dev] [FD.io Helpdesk #41921] connection interruptiones >> between jenkins executor and VIRL servers >> >> Hello Anton, >> >> Thanks for the fast response. We will check local firewall setting as you >> proposed. >> >> Regards, >> Jan >> >> -----Original Message----- >> From: Anton Baranov via RT [mailto:fdio-helpd...@rt.linuxfoundation.org] >> Sent: Tuesday, June 20, 2017 17:13 >> To: Jan Gelety -X (jgelety - PANTHEON TECHNOLOGIES at Cisco) >> <jgel...@cisco.com> >> Cc: csit-...@lists.fd.io; vpp-dev@lists.fd.io >> Subject: [FD.io Helpdesk #41921] connection interruptiones between jenkins >> executor and VIRL servers >> >> Jan: >> >> This is what I got from fdio jenkins server (i did the tests with >> 10.30.{52,53}.2 hosts: >> >> $ ip ro get 10.30.52.2 >> 10.30.52.2 via 10.30.48.1 dev eth0 src 10.30.48.5 >> cache >> >> The traffic is going directly through neutron router.. so we don't block any >> traffic on our firewall >> >> $ ping -q -c4 10.30.52.2 >> PING 10.30.52.2 (10.30.52.2) 56(84) bytes of data. >> >> --- 10.30.52.2 ping statistics --- >> 4 packets transmitted, 4 received, 0% packet loss, time 3001ms rtt >> min/avg/max/mdev = 0.496/0.789/1.509/0.419 ms >> >> I was able to reach the host in 10.30.52.0/24 network from jenkins server >> >> $ nc -nv 10.30.52.2 22 >> Ncat: Version 6.40 ( http://nmap.org/ncat ) >> Ncat: Connection refused. >> >> Looks like access is blocked there. Could you check your local firewall >> setting and make sure you allow port 22/tcp ? >> >> The above is also true for 10.30.{53,54}.0/24 subnets >> >> Regards, >> >> On Tue Jun 20 10:51:15 2017, jgel...@cisco.com wrote: >>> Hello Vanessa, >>> >>> Thanks for the info. >>> >>> Just few remarks: >>> >>> 1. virl1 (10.30.51.28) - nodes of simulations started there are using >>> subnet 10.30.52.0/24 and we are experiencing ssh timeouts in this >>> subnet >>> >>> 2. virl2 (10.30.51.29) - nodes of simulations started there were >>> using subnet 10.30.53.0/24 and we were experiencing ssh timeouts in >>> this subnet; >>> - at the moment we switched the >>> subnet back to 10.30.51.0/24 and assigned there IP pool 10.30.51.106 - >>> 10.30.51.180 >>> - new tests started - let you >>> know the result tomorrow >>> >>> 3. virl3 (10.30.51.30) - nodes of simulations started there were using >>> subnet 10.30.51.0/24 and IP pool is set to 10.30.51.181 - 10.30.51.254 >>> and we didn't experience ssh timeouts in this subnet; >>> >>> >>> So would it be possible to check routes for subnets 10.30.52.0/24, >>> 10.30.53.0/24 and also for 10.30.54.0/24 (that is planned for vilr3 >>> when it will be upgraded)? >>> >>> Thank you very much. >>> >>> Regards, >>> Jan >>> >>> -----Original Message----- >>> From: Vanessa Valderrama via RT [mailto:fdio- >>> helpd...@rt.linuxfoundation.org] >>> Sent: Friday, June 16, 2017 22:01 >>> To: Jan Gelety -X (jgelety - PANTHEON TECHNOLOGIES at Cisco) >>> <jgel...@cisco.com> >>> Cc: csit-...@lists.fd.io; vpp-dev@lists.fd.io >>> Subject: [FD.io Helpdesk #41921] connection interruptiones between >>> jenkins executor and VIRL servers >>> >>> We did have the vendor run MTRs. I've attached the results. >>> >>> On Fri Jun 16 15:36:16 2017, valderrv wrote: >>>> Jan, >>>> >>>> I missed this conversation with abranov. Can this issue be resolved? >>>> >>>> <snip> >>>> <jgelety> abranov: unfortunatley I had no time to check test logs >>>> from last test cases before (because of meeting so I just had a look >>>> to console output) and I found out that ssh failures are not related >>>> to connection between jenkins and virl now (but it was this issue at >>>> the time I wrote the e-mail). >>>> <mnaser> [11:04:06] <jgelety> They are related to start up pf >>>> nested VM now - so I will ask VIRL support for the help here. >>>> </snip> >>>> >>>> Thank you, >>>> Vanessa >>>> >>>> On Fri Jun 16 12:12:43 2017, valderrv wrote: >>>>> Jan, >>>>> >>>>> We are looking into this issue. >>>>> >>>>> Thank you, >>>>> Vanessa >>>>> >>>>> On Fri Jun 16 09:12:55 2017, jgel...@cisco.com wrote: >>>>>> Hello Anton, >>>>>> >>>>>> Unfortunately we are still having issues with ssh connection >>>>>> timeouts during tests on virl. Could you, please, have a look on >>>>>> it? >>>>>> >>>>>> Thank you very much. >>>>>> Regards, >>>>>> Jan >>>>>> >>>>>> -----Original Message----- >>>>>> From: Anton Baranov via RT [mailto:fdio- >>>>>> helpd...@rt.linuxfoundation.org] >>>>>> Sent: Wednesday, June 14, 2017 15:45 >>>>>> To: Jan Gelety -X (jgelety - PANTHEON TECHNOLOGIES at Cisco) >>>>>> <jgel...@cisco.com> >>>>>> Cc: csit-...@lists.fd.io; vpp-dev@lists.fd.io >>>>>> Subject: [FD.io Helpdesk #41921] connection interruptiones >>>>>> between jenkins executor and VIRL servers >>>>>> >>>>>> Jan: >>>>>> >>>>>> On my side I currently don't see any connectivity problems >>>>>> between jenkins and VIRL servers. Please let me know if you're >>>>>> still having that issue. I'll keep an eye on that problem and if >>>>>> it reapears I'll report that to our cloud provider to check >>>>>> further. >>>>>> >>>>>> Thanks, >>>>>> -- >>>>>> Anton Baranov >>>>>> Systems and Network Administrator The Linux Foundation >>>>>> >>>>>> On Wed Jun 14 08:12:45 2017, jgel...@cisco.com wrote: >>>>>>> Dear held...@fd.io<mailto:held...@fd.io> >>>>>>> >>>>>>> We are observing connection issues between Jenkins executors >>>>>>> and VIRL servers that leads to failures of verify jobs >>>>>>> (https://jenkins.fd.io/view/vpp/job/vpp-csit-verify-virl- >>>>>>> master/ >>>>>>> , >>>>>>> https://jenkins.fd.io/view/csit/job/csit-vpp-functional-master >>>>>>> - ubuntu1604-virl/, >>>>>>> https://jenkins.fd.io/view/csit/job/csit-vpp- >>>>>>> functional-master-centos7-virl/) because of ssh connection >>>>>>> timeouts. >>>>>>> >>>>>>> Could you, please, have a look on it? >>>>>>> >>>>>>> Thank you very much. >>>>>>> >>>>>>> Regards, >>>>>>> Jan >>>>>> >>>>>> >>>>>> >>>>> >>>>> >>> >>> >>> >> >> >> -- >> Anton Baranov >> Systems and Network Administrator >> The Linux Foundation >> >> _______________________________________________ >> vpp-dev mailing list >> vpp-dev@lists.fd.io >> https://lists.fd.io/mailman/listinfo/vpp-dev >> _______________________________________________ >> csit-dev mailing list >> csit-...@lists.fd.io >> https://lists.fd.io/mailman/listinfo/csit-dev > _______________________________________________ vpp-dev mailing list vpp-dev@lists.fd.io https://lists.fd.io/mailman/listinfo/vpp-dev
