Hi Sridhar and all, I may be a few minutes late to our meeting today, it all depends on how long my doctor's appointment runs-over, but I'll join you as soon as I can.
FYI - I had an action to create a JIRA ticket for TST009 review: https://jira.opnfv.org/browse/VSPERF-552 see you soon, Al From: Klozik, MartinX [mailto:[email protected]] Sent: Wednesday, December 13, 2017 7:43 AM To: Rao, Sridhar; [email protected] Cc: MORTON, ALFRED C (AL); Cooper, Trevor; 'Mars Toktonaliev (Nokia - US/Irving)'; Ferriter, Cian; Christian Trautman ([email protected]); Bill Michalowski ([email protected]); Elias, RichardX Subject: RE: [vsperf] Agenda for VSPERF weekly meeting - 13 Dec 2017 (ww49) Hi All, I won't be able to join today. During last weeks I was trying to fix our CI jobs, which was a bit more challenging than expected. There are two separate issues: 1) VERIFY & MERGE jobs - there are issues with execution of VPP testcases at ericcson-build3 machine. Thus some time ago I changed configuration of VERIFY & MERGE jobs to be executed at ericsson-build4 machine. For some time it worked, but then VPP has failed and it is not working there at all. In order to be able to pass gerrit verification of new patches, I had to disable VPP tests - it means that if vppctl is not able to connect to running VPP, then VPP tests are skipped to avoid verification failure. As I'm not able to connect to this server I can't find the root cause. The easiest "solution" is to reboot that server. I've asked responsible admins for help some time ago, but I've not heard from them for some time now. So I tried another approach, it means to allow execution of VERIFY & MERGE jobs at our intel-pod12. I've changed priorities, so Jenkins would firstly check if pod12 is idle with fallback to ericsson-build4 machine. Patch is available at gerrit and waiting for review from releng team: https://gerrit.opnfv.org/gerrit/#/c/47581/<https://urldefense.proofpoint.com/v2/url?u=https-3A__gerrit.opnfv.org_gerrit_-23_c_47581_&d=DwQFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=dni7PJlv6kNbMGrbHV3yFYFdyrZkUJs1iHQjXOx3a_s&s=8-VSlOsazD_vfx59RhB6NyWCuhccW0gWwo5E4NF7RHg&e=> 2) DAILY job execution always failed after POD12 was moved to the new location; The reason was crash and reboot of node2 during execution of Ovs Vanilla testcases. It took me a while to find a root cause, which is actually a kernel crash during removal of "openvswitch.ko" kernel module. Originally I though, that it is caused by some resources allocated by OVS, which were not cleared properly after OVS shutdown. However it happens also in case, that OVS is not started at all. I was able to reproduce it on node2 (CentOS7.3) just by calling insmod and rmmod on openvswitch kernel module several times. Sometimes machine crashes after a few attempts, other time it takes more than 10 iterations of insmod/rmmod. I tried 2 different 3.10.0-xxx kernel versions and two different OVS versions with the same output. It seems, that it is specific to RHT's patched 3.10 kernel. In case that the same OVS version is used with 4.x kernel, then system is stable (I was able to run several dozens of insmod/rmmod iterations without any issue). Thus I've installed kernel-lt from elrepo.org to node2, to be able to run our daily job successfully. With 4.4 kernel, I was able to execute both vswitchperf-daily-master and vswitchperf-daily-euphrates several times. Unfortunately it has failed again recently, but this time during execution of VPP TCs. One time VPP was not able to acquire one NIC. Second time boot of VM was very slow and compilation of testpmd took over 30s (it normally takes about 8s). Let's see if these incidents will happen more often or not. Also some performance figures are varying. For example pvp_tput executed manually at node2 measures over 11 mil. FPS, but during daily job execution, it often measures about 7 or 8. This needs some further investigation to find the root cause. Best Regards, Martin From: Rao, Sridhar [mailto:[email protected]] Sent: Wednesday, December 13, 2017 6:22 AM To: [email protected]<mailto:[email protected]> Cc: 'ALFRED C 'MORTON (AL)'' <[email protected]<mailto:[email protected]>>; Cooper, Trevor <[email protected]<mailto:[email protected]>>; 'Mars Toktonaliev (Nokia - US/Irving)' <[email protected]<mailto:[email protected]>>; Ferriter, Cian <[email protected]<mailto:[email protected]>>; Klozik, MartinX <[email protected]<mailto:[email protected]>>; Christian Trautman ([email protected]<mailto:[email protected]>) <[email protected]<mailto:[email protected]>>; Bill Michalowski ([email protected]<mailto:[email protected]>) <[email protected]<mailto:[email protected]>>; Elias, RichardX <[email protected]<mailto:[email protected]>> Subject: [vsperf] Agenda for VSPERF weekly meeting - 13 Dec 2017 (ww49) Hi All, The agenda for this week: 1. Development update - Review of the current status of release plan. 2. Plugfest updates. Meeting minutes * WW48 http://ircbot.wl.linuxfoundation.org/meetings/opnfv-vswitchperf/2017/opnfv-vswitchperf.2017-11-29-16.14.html<https://urldefense.proofpoint.com/v2/url?u=http-3A__ircbot.wl.linuxfoundation.org_meetings_opnfv-2Dvswitchperf_2017_opnfv-2Dvswitchperf.2017-2D11-2D29-2D16.14.html&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=dni7PJlv6kNbMGrbHV3yFYFdyrZkUJs1iHQjXOx3a_s&s=fQrj3xTmGchplAyy_ekIXdVNrraVHTtjPeu0j_qbNvY&e=> * WW47 http://ircbot.wl.linuxfoundation.org/meetings/opnfv-vswitchperf/2017/opnfv-vswitchperf.2017-11-22-16.02.html<https://urldefense.proofpoint.com/v2/url?u=http-3A__ircbot.wl.linuxfoundation.org_meetings_opnfv-2Dvswitchperf_2017_opnfv-2Dvswitchperf.2017-2D11-2D22-2D16.02.html&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=dni7PJlv6kNbMGrbHV3yFYFdyrZkUJs1iHQjXOx3a_s&s=ZMA92aQ5gtf7w70vx-a3woVZcXtthbDhbQdqbWN7f68&e=> * WW46 http://ircbot.wl.linuxfoundation.org/meetings/opnfv-vswitchperf/2017/opnfv-vswitchperf.2017-11-15-16.01.html<https://urldefense.proofpoint.com/v2/url?u=http-3A__ircbot.wl.linuxfoundation.org_meetings_opnfv-2Dvswitchperf_2017_opnfv-2Dvswitchperf.2017-2D11-2D15-2D16.01.html&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=dni7PJlv6kNbMGrbHV3yFYFdyrZkUJs1iHQjXOx3a_s&s=0iA2lXN5d0iNhXeO-M4Gm9UrpQ-pHorZhYwIOytqaVE&e=> -------------------------------------------------------- Time: Wednesday UTC 16h00, Ireland (UTC) 16h00, PST 8h00 (UTC/GMT -8) IRC: freenode https://freenode.net/<https://urldefense.proofpoint.com/v2/url?u=https-3A__freenode.net_&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=dni7PJlv6kNbMGrbHV3yFYFdyrZkUJs1iHQjXOx3a_s&s=CnniAPUFH1Um_ZMOTi4AgH9QGPiFTDgoX_wGGcHIPJ4&e=> channel: #opnfv-vswitchperf http://webchat.freenode.net/?channels=opnfv-vswitchperf<https://urldefense.proofpoint.com/v2/url?u=http-3A__webchat.freenode.net_-3Fchannels-3Dopnfv-2Dvswitchperf&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=dni7PJlv6kNbMGrbHV3yFYFdyrZkUJs1iHQjXOx3a_s&s=wsyiljre6W9BsFgcVFddV6tqtcEn-qn3tSBe7YGImrw&e=> Audio: https://global.gotomeeting.com/join/391235029<https://urldefense.proofpoint.com/v2/url?u=https-3A__global.gotomeeting.com_join_391235029&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=dni7PJlv6kNbMGrbHV3yFYFdyrZkUJs1iHQjXOx3a_s&s=mAxwYeFGcacCf3bal8IB_AHZi38HQgKPnN-ZdR-qFeM&e=> You can also dial in using your phone. United States: +1 (571) 317-3116 Access Code: 391-235-029 Phone numbers from other countries are here ... https://wiki.opnfv.org/display/meetings<https://urldefense.proofpoint.com/v2/url?u=https-3A__wiki.opnfv.org_display_meetings&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=dni7PJlv6kNbMGrbHV3yFYFdyrZkUJs1iHQjXOx3a_s&s=-mrrKxGjSCiuMp-vgEYwQH3TvfZd0QafNHLw6vY1eZE&e=> Regards, Sridhar K. N. Rao (Ph. D) Architect +91-9900088064 Spirent Communications e-mail confidentiality. ------------------------------------------------------------------------ This e-mail contains confidential and / or privileged information belonging to Spirent Communications plc, its affiliates and / or subsidiaries. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution and / or the taking of any action based upon reliance on the contents of this transmission is strictly forbidden. If you have received this message in error please notify the sender by return e-mail and delete it from your system. Spirent Communications plc Northwood Park, Gatwick Road, Crawley, West Sussex, RH10 9XN, United Kingdom. Tel No. +44 (0) 1293 767676 Fax No. +44 (0) 1293 767677 Registered in England Number 470893 Registered at Northwood Park, Gatwick Road, Crawley, West Sussex, RH10 9XN, United Kingdom. Or if within the US, Spirent Communications, 27349 Agoura Road, Calabasas, CA, 91301, USA. Tel No. 1-818-676- 2300 -------------------------------------------------------------- Intel Research and Development Ireland Limited Registered in Ireland Registered Office: Collinstown Industrial Park, Leixlip, County Kildare Registered Number: 308263 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.
_______________________________________________ opnfv-tech-discuss mailing list [email protected] https://lists.opnfv.org/mailman/listinfo/opnfv-tech-discuss
