update: mnaser (vex host ) did a ‘live migration’ of the jenkins.fd.io<http://jenkins.fd.io> ’server’ taking RTT from >6 seconds back down to the @100ms range. As a result jobs are starting up again and the queue is grinding through the backlog and new jobs.
Im not currently seeing the same delay on/towards other hosted services gerrit/nexus etc. Ill be watching it closely the next few hours but please let me know if you see any unusual errors. thanks, Ed On Feb 27, 2019, at 8:34 AM, Ed Kern via Lists.Fd.Io <ejk=cisco....@lists.fd.io<mailto:ejk=cisco....@lists.fd.io>> wrote: fyi to vpp-dev as well.. debugging with vexxhost Begin forwarded message: From: "Peter Mikus via Lists.Fd.Io" <pmikus=cisco....@lists.fd.io<mailto:pmikus=cisco....@lists.fd.io>> Subject: [csit-dev] Jenkins infra outage - 100% jobs queued, VPN access very slow Date: February 27, 2019 at 8:32:50 AM MST To: "helpd...@fd.io<mailto:helpd...@fd.io>" <helpd...@fd.io<mailto:helpd...@fd.io>> Cc: csit-...@lists.fd.io<mailto:csit-...@lists.fd.io> Reply-To: pmi...@cisco.com<mailto:pmi...@cisco.com> Hello team, We are observing jobs queueing in Jenkins.fd.io<http://jenkins.fd.io/> and also the VPN access to infra is very slow. Can you please investigate? ping 10.30.51.50 PING 10.30.51.50 (10.30.51.50) 56(84) bytes of data. 64 bytes from 10.30.51.50: icmp_seq=1 ttl=62 time=9961 ms 64 bytes from 10.30.51.50: icmp_seq=2 ttl=62 time=8928 ms 64 bytes from 10.30.51.50: icmp_seq=3 ttl=62 time=7907 ms 64 bytes from 10.30.51.50: icmp_seq=4 ttl=62 time=6887 ms 64 bytes from 10.30.51.50: icmp_seq=5 ttl=62 time=5863 ms 64 bytes from 10.30.51.50: icmp_seq=6 ttl=62 time=4834 ms 64 bytes from 10.30.51.50: icmp_seq=7 ttl=62 time=3809 ms 64 bytes from 10.30.51.50: icmp_seq=8 ttl=62 time=2790 ms 64 bytes from 10.30.51.50: icmp_seq=9 ttl=62 time=1762 ms Thank you. Peter Mikus Engineer – Software Cisco Systems Limited -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#3348): https://lists.fd.io/g/csit-dev/message/3348 Mute This Topic: https://lists.fd.io/mt/30153144/675649 Group Owner: csit-dev+ow...@lists.fd.io<mailto:csit-dev+ow...@lists.fd.io> Unsubscribe: https://lists.fd.io/g/csit-dev/unsub [e...@cisco.com<mailto:e...@cisco.com>] -=-=-=-=-=-=-=-=-=-=-=- -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#12363): https://lists.fd.io/g/vpp-dev/message/12363 Mute This Topic: https://lists.fd.io/mt/30153159/675649 Group Owner: vpp-dev+ow...@lists.fd.io<mailto:vpp-dev+ow...@lists.fd.io> Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [e...@cisco.com<mailto:e...@cisco.com>] -=-=-=-=-=-=-=-=-=-=-=-
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#12364): https://lists.fd.io/g/vpp-dev/message/12364 Mute This Topic: https://lists.fd.io/mt/30153159/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
