+ hc2vpp (all hc2vpp csit jobs affected) Marek
From: vpp-dev@lists.fd.io [mailto:vpp-dev@lists.fd.io] On Behalf Of Maciek Konstantynowicz (mkonstan) Sent: 6 marca 2018 15:55 To: vpp-dev@lists.fd.io Cc: csit-...@lists.fd.io Subject: [vpp-dev] number of csit-vpp jobs failing on vpp master branch - RCA wip Heads-up - resolution in progress.. -Maciek Begin forwarded message: From: Maciek Konstantynowicz <mkons...@cisco.com<mailto:mkons...@cisco.com>> Subject: number of csit-vpp jobs failing on vpp master branch - RCA wip Date: 6 March 2018 at 14:54:39 GMT To: csit-...@lists.fd.io<mailto:csit-...@lists.fd.io> Cc: Peter Mikus <pmi...@cisco.com<mailto:pmi...@cisco.com>> RCA Update from Peter Mikus on two issues behind the failing csit-vpp jobs due to recent VPP patches:- Side note - honeycomb jobs are also affected. 1. errors during VPP install due to new vpp-lib dependencies - observed from last Friday Root cause ---------- VPP patch [1] introduce new external package dependencies (libmbedcrypto0 libmbedtls10 libmbedx509-0). Without these packages VPP service is not starting results in following log messages: vpp[30760]: /usr/bin/vpp: error while loading shared libraries: libmbedtls.so.10: cannot open shared object file: No such file or directory This was not caught by vpp-csit-verify-virl-master job due to the bug in evaluating the robot framework results from multiple parallel simulation (see full log in [2]). Root cause is error in writing into /run/shm/rc_test_run${N} where CSIT bootstrap.sh is storing the results of partial runs. This results into faulty evaluation of total number of passed/failed test cases. There are several steps needed to fix this issue: #1 - Fix bootstrap to proper evaluate the results https://gerrit.fd.io/r/#/c/10981/ #2 – Install the missing dependencies in VIRL Ubuntu image .. TODO (patch https://gerrit.fd.io/r/#/c/10980/) - Rebuild of Ubuntu image needed #3 – Install the missing dependencies in VIRL Centos image .. TODO (patch https://gerrit.fd.io/r/#/c/10980/) - Rebuild of Ubuntu image needed #4 – Install the missing dependencies on PhyTB .. DONE (patch https://gerrit.fd.io/r/#/c/10980/) [1] https://gerrit.fd.io/r/10768 [2] https://logs.fd.io/production/vex-yul-rot-jenkins-1/vpp-csit-verify-virl-master/9755/console.log.gz 2. startup.conf broken - observed from last Friday impacting all csit-vpp jobs and hc-vpp jobs wasn't detected in vpp-csit-verify Root cause ---------- This issue is caused by api-segment { gid vpp } to be not recognized in default startup.conf installed by VPP, resulting into VPP service not started. Root cause to be confirmed. Work in progress. Need to fix the vpp-lib dependencies first.
