Hi Vanessa, Agreed, the failure pattern does not exactly point to the new instances. However, at least three jobs [1-3] ran for many hours and eventually failed in ‘unexpected’ ways (e.g., timeout and python crashes). Thereby, my suspicion was that it may have to do with some form of starvation. I hope the performance instances will help with that.
I’ve rebased 7108, let’s see if things get back to normal. Thanks for the help! Florin [1] https://jenkins.fd.io/job/vpp-verify-master-ubuntu1604/7098/console <https://jenkins.fd.io/job/vpp-verify-master-ubuntu1604/7098/console> [2] https://jenkins.fd.io/job/vpp-verify-master-ubuntu1604/7102/console <https://jenkins.fd.io/job/vpp-verify-master-ubuntu1604/7102/console> [3] https://jenkins.fd.io/job/vpp-verify-master-ubuntu1604/7108/console <https://jenkins.fd.io/job/vpp-verify-master-ubuntu1604/7108/console> > On Sep 14, 2017, at 3:59 PM, Vanessa Valderrama via RT > <fdio-helpd...@rt.linuxfoundation.org> wrote: > > Florin, > > It doesn't appear the failures are related to the new instances. We > will be switching to a new performance (4c/8GB) instance when it's > available. I've submitted a request to the vendor and will follow up > tomorrow morning. Adding the additional CPU resources may > improve/resolve the failure in build 7108. Please let me know if there > are additional failures you are concerned about that I haven't > addressed. Also please feel free to ping me via IRC fdio-infra > (valderrv) if you'd like to work together to debug these issues quickly. > > *https://jenkins.fd.io/view/vpp/job/vpp-verify-master-ubuntu1604/7108/console > 17:34:21* ACL IPv4 routed -> bridged, L2 ACL > permit+reflect17:44:47,177 Timeout while waiting for child test runner > process (last test running was `MACIP 2 ACLs each with 100+ entries' in > `/tmp/vpp-unittest-TestMACIP-py7viS')! > > * > *This failure is not a timeout and appears to be related to code* > > https://jenkins.fd.io/view/vpp/job/vpp-verify-master-ubuntu1604/7109/console* > > > * 16:48:11* * Test framework PEP8 compliance check FAILED > *16:48:11* > ******************************************************************* > *16:48:11* Makefile:172: recipe for target 'checkstyle' failed > *16:48:11* make[1]: *** [checkstyle] > Error 1 *16:48:11* make[1]: Leaving directory > '/w/workspace/vpp-verify-master-ubuntu1604/test' *16:48:11* > Makefile:376: recipe for target 'test-checkstyle' failed *16:48:11* > make: *** [test-checkstyle] Error 2 *16:48:11* Build step > 'Execute shell' marked build as failure > > > The cause of these two failures appear to be related to package(s) not > installing? > * > https://jenkins.fd.io/view/vpp/job/vpp-verify-master-ubuntu1604/7111/console > > https://jenkins.fd.io/view/vpp/job/vpp-verify-master-ubuntu1604/7112/console > ** 19:05:38* cc1: all warnings being treated as errors *19:05:38* > Makefile:6164: recipe for target 'vppinfra/linux/mem.lo' failed > *19:05:38* make[3]: *** [vppinfra/linux/mem.lo] Error 1 *19:05:38* > make[3]: *** Waiting for unfinished jobs.... * > 19:05:40* make[3]: Leaving directory > '/w/workspace/vpp-verify-master-ubuntu1604/build-root/build-vpp-native/vpp'*19:05:40* > Makefile:698: recipe for target 'vpp-build' failed *19:05:40* make[2]: > *** [vpp-build] Error 2 *19:05:40* make[2]: > Leaving directory > '/w/workspace/vpp-verify-master-ubuntu1604/build-root' *19:05:40* > Makefile:934: recipe for target 'install-packages' failed *19:05:40* > make[1]: *** [install-packages] Error 1 *19:05:40* make[1]: Leaving > directory '/w/workspace > /vpp-verify-master-ubuntu1604/build-root' *19:05:40* Makefile:487: > recipe for target 'verify' failed *19:05:40* make: *** [verify] Error 2 > *19:05:40* Build step 'Execute shell' marked build as failure > > Thank you, > Vanessa > > On 09/14/2017 02:27 PM, Florin Coras wrote: >> Hi Vanessa, >> >> We’re seeing lots of vpp-verify-master-ubuntu1604 failures that seem to be >> timeouts related. Could they be related to the recent switch to the new >> instances? >> >> Thanks, >> Florin >> > > _______________________________________________ vpp-dev mailing list vpp-dev@lists.fd.io https://lists.fd.io/mailman/listinfo/vpp-dev
