Folks,
After troubleshooting latency issues in the datapath between the Jenkins
openstack instance and the Nomad cluster, the host of the Ingress
instance appeared to be the source of the problem and the Ingress
instance was live migrated to another host. In addition, the primary
Nomad server was rebooted. These changes resolved the 'Java Connection
Closed Exception' issues.
However, at this time, the vpp-device job is still failing due to a
known issue which will be resolved when Peter Mikus comes online
tomorrow morning CET. Once the vpp-device job failures have been
resolved, I will be issuing 'recheck' on open VPP gerrit changes which
are failing due to the vpp-device job. Please feel free to 'recheck'
your gerrit changes if you would like to verify that the rest of the CI
jobs complete successfully.
I'd like to thank Mohammed Naser, Vanessa Valderrama, Anton Baranov,
Peter Mikus & Maciek Konstantynowicz for their coordinated efforts in
resolving this outage.
Thanks again for your patience during this CI outage.
-daw-
On 7/19/2021 10:51 PM, Dave Wallace via lists.fd.io wrote:
Folks,
Vanessa performed a Jenkins reset at my request to see if that would
resolve this problem. Unfortunately the Jenkins reset did not resolve
the connection resets. A recheck of gerrit change after the Jenkins
restart failed with multiple job failures due to TCP connection resets:
https://gerrit.fd.io/r/c/vpp/+/32858/6#message-c77806c2fd58c3c00935e1b5589a402e4b670f9f
There has also been no correlation with Ping Monitor events, Nomad
cluster events, Nomad host, subnet, or docker image.
Investigation continues in the datapath between the Jenkins openstack
instance and the Nomad cluster.
Thanks again for your patience.
-daw-
On 7/19/2021 11:29 AM, Dave Wallace via lists.fd.io wrote:
Folks,
There have been large numbers CI job failures due to 'Java Connection
Closed Exception' that appear to have started occurring on July 17.
I have opened a ticket with Vexxhost and am actively diagnosing the
problem with them.
Thank you for your patience while the issue is being resolved.
-daw-
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#19840): https://lists.fd.io/g/vpp-dev/message/19840
Mute This Topic: https://lists.fd.io/mt/84310372/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
