Hello, CRITICAL ISSUE: ALL CSIT PERFORMANCE TESTING JOBS ARE EFFECTED. 1904 RLS SHOWSTOPPER.
This is now a critical issue as all CSIT performance testing jobs are failing, preventing FD.io<http://FD.io> CSIT project to complete the CSIT-1904 release. This ticket #74794, is related to the same underlying jnlp connectivity issue reported in other recent tickets #74795, #73486, #73487, but apparently issue was observed earlier #70022, and much earlier too.. It was attributed to haproxy issues (Layer4 timeouts), but the issue is still happening. Below is the sorted list of timestamps from Peter Mikus script as per earlier note in this case. One can see that multiple jobs get affected at the same time. Please advise on resolution, and if any assistance is needed from FD.io<http://FD.io> CSIT project. Thank you for your attention! Cheers, -Maciek 2019.04.27-10:54:40 java.nio.channels.ClosedChannelException 2019.04.27-10:55:06 java.nio.channels.ClosedChannelException 2019.04.27-10:55:10 java.nio.channels.ClosedChannelException 2019.04.28-09:32:20 java.nio.channels.ClosedChannelException 2019.04.30-16:16:29 java.nio.channels.ClosedChannelException 2019.04.30-16:16:29 java.nio.channels.ClosedChannelException 2019.04.30-16:16:29 java.nio.channels.ClosedChannelException 2019.04.30-16:16:29 java.nio.channels.ClosedChannelException 2019.04.30-16:16:29 java.nio.channels.ClosedChannelException 2019.04.30-16:16:29 java.nio.channels.ClosedChannelException 2019.04.30-16:16:29 java.nio.channels.ClosedChannelException 2019.05.01-13:48:18 java.nio.channels.ClosedChannelException 2019.05.01-13:48:18 java.nio.channels.ClosedChannelException 2019.05.01-13:48:18 java.nio.channels.ClosedChannelException 2019.05.01-13:48:26 java.nio.channels.ClosedChannelException 2019.05.01-13:48:26 java.nio.channels.ClosedChannelException 2019.05.01-13:48:26 java.nio.channels.ClosedChannelException 2019.05.01-13:48:26 java.nio.channels.ClosedChannelException 2019.05.01-13:48:26 java.nio.channels.ClosedChannelException 2019.05.01-13:48:26 java.nio.channels.ClosedChannelException 2019.05.01-13:48:26 java.nio.channels.ClosedChannelException 2019.05.01-13:48:27 java.nio.channels.ClosedChannelException 2019.05.01-17:59:20 java.nio.channels.ClosedChannelException On 2 May 2019, at 07:47, Peter Mikus -X (pmikus - PANTHEON TECHNOLOGIES at Cisco) <pmi...@cisco.com<mailto:pmi...@cisco.com>> wrote: Hello, This issue is starting to be very serious and impacting deliverables for CSIT project. Can you please investigate with priority. Subset of failed jobs (just few examples from past days): https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-2n-skx/38/ https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-2n-skx/37/ https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-2n-skx/36/ https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-2n-skx/35/ https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-2n-skx/34/ https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-3n-skx/20/ https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-3n-skx/19/ https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-3n-skx/18/ https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-3n-skx/17/ https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-3n-skx/15/ https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-3n-skx/13/ https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-3n-hsw/16/ https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-3n-hsw/17/ https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-3n-hsw/18/ https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-3n-hsw/19/ https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-3n-hsw/10/ https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-3n-hsw/13/ https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-3n-hsw/14/ https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-3n-hsw/15/ https://jenkins.fd.io/job/csit-dpdk-perf-verify-1904-3n-skx/8/ https://jenkins.fd.io/job/csit-dpdk-perf-verify-1904-3n-skx/10/ https://jenkins.fd.io/job/csit-vpp-perf-mrr-daily-master/666/ https://jenkins.fd.io/job/csit-vpp-perf-mrr-daily-master/665/ Thank you. Note: From logs, it can be quite easily extracted dates and times of outages and correlation in time. jobs=('https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-2n-skx/38/' 'https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-2n-skx/37/' 'https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-2n-skx/36/' 'https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-2n-skx/35/' 'https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-2n-skx/34/' 'https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-3n-skx/20/' 'https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-3n-skx/19/' 'https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-3n-skx/18/' 'https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-3n-skx/17/' 'https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-3n-skx/15/' 'https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-3n-skx/13/' 'https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-3n-hsw/16/' 'https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-3n-hsw/17/' 'https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-3n-hsw/18/' 'https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-3n-hsw/19/' 'https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-3n-hsw/10/' 'https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-3n-hsw/13/' 'https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-3n-hsw/14/' 'https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-3n-hsw/15/' 'https://jenkins.fd.io/job/csit-dpdk-perf-verify-1904-3n-skx/8/' 'https://jenkins.fd.io/job/csit-dpdk-perf-verify-1904-3n-skx/10/' 'https://jenkins.fd.io/job/csit-vpp-perf-mrr-daily-master/666/' 'https://jenkins.fd.io/job/csit-vpp-perf-mrr-daily-master/665/' ' https://jenkins.fd.io/job/csit-vpp-perf-mrr-daily-master-3n-skx/416/') for job in "${jobs[@]}"; do curl -s "${job}/timestamps/?time=yyyy.MM.dd-HH:mm:ss&appendLog&locale=en_US" | grep -m 1 "java.nio.channels.ClosedChannelException" done which results in: 2019.05.01-13:48:27 java.nio.channels.ClosedChannelException 2019.05.01-13:48:26 java.nio.channels.ClosedChannelException 2019.05.01-13:48:26 java.nio.channels.ClosedChannelException 2019.05.01-13:48:18 java.nio.channels.ClosedChannelException 2019.04.30-16:16:29 java.nio.channels.ClosedChannelException 2019.05.01-13:48:18 java.nio.channels.ClosedChannelException 2019.05.01-13:48:26 java.nio.channels.ClosedChannelException 2019.05.01-13:48:26 java.nio.channels.ClosedChannelException 2019.04.30-16:16:29 java.nio.channels.ClosedChannelException 2019.04.30-16:16:29 java.nio.channels.ClosedChannelException 2019.04.27-10:55:06 java.nio.channels.ClosedChannelException 2019.04.30-16:16:29 java.nio.channels.ClosedChannelException 2019.05.01-13:48:26 java.nio.channels.ClosedChannelException 2019.05.01-17:59:20 java.nio.channels.ClosedChannelException 2019.05.01-13:48:26 java.nio.channels.ClosedChannelException 2019.04.27-10:54:40 java.nio.channels.ClosedChannelException 2019.04.28-09:32:20 java.nio.channels.ClosedChannelException 2019.04.30-16:16:29 java.nio.channels.ClosedChannelException 2019.04.30-16:16:29 java.nio.channels.ClosedChannelException 2019.05.01-13:48:18 java.nio.channels.ClosedChannelException 2019.05.01-13:48:26 java.nio.channels.ClosedChannelException 2019.04.30-16:16:29 java.nio.channels.ClosedChannelException 2019.04.27-10:55:10 java.nio.channels.ClosedChannelException Peter Mikus Engineer – Software Cisco Systems Limited From: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> <vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io>> On Behalf Of Jan Gelety via Lists.Fd.Io Sent: Monday, April 29, 2019 8:58 AM To: helpd...@fd.io<mailto:helpd...@fd.io>; fdio-helpd...@rt.linuxfoundation.org<mailto:fdio-helpd...@rt.linuxfoundation.org> Cc: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> Subject: [vpp-dev] Jenkins.fd.io network issues Hello, We are experiencing quite a lot of network issues when running CSIT tests for 19.04 report: Caused: hudson.remoting.ChannelClosedException: Channel "unknown": Remote call on JNLP4-connect connection from vex-yul-rot-ingress-1.ci.codeaurora.org/10.30.48.3:41068<http://vex-yul-rot-ingress-1.ci.codeaurora.org/10.30.48.3:41068> failed. The channel is closing down or has closed down https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-3n-hsw/13/console Could you, please, have a look on it? Thank you very much. Regards, Jan
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#12908): https://lists.fd.io/g/vpp-dev/message/12908 Mute This Topic: https://lists.fd.io/mt/31454819/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
