On Mon, 2018-09-10 at 14:29 +0000, Ed Kern (ejk) wrote: > At least three different possible actions to take at this point: (outside of > fixing the issue) > > > > > 1. remove make test attempt from arm build (return it to the way it was before > a week ago). > > 2. lower timeout further my first thought would be in the 75 minute range > (from 120) I don't like this option mainly because it would still imply that developers will have to wait for 75 minutes to see a patch verified... The worst is that the job is a non-voting one hence having it running does add any meaningful insight into the build result. I'm in favour for the issue being resolved in the sandbox and eventually moved to production when stable. > 3. remove job altogether. I believe this should be the option to purse unless it could be fixed quickly.As it is it only causes delays to the overall build / review / merging process... > > > Im happy to push a patch that accomplishes the above or other options I > haven’t thought of > for this mail. > > > > Just let me know.. > > > > Ed > > > > > > > > > > On Sep 10, 2018, at 4:31 AM, Marco Varlese <mvarl...@suse.de> wrote: > > > > Last > > example (today) can be found here > > > > > > > > 10:47:06 > > Not running extended tests (some tests will be skipped) > > > > 10:47:06 > > Perform 4 attempts to pass the suite... > > > > 10:47:10 > > *** Error in `python': double free or corruption (out): > > > > 0x0000ffff75d483f0 > > *** > > > > 12:27:54 > > Build timed out (after 120 minutes). Marking the build as failed. > > > > > > > > Patch: https://gerrit.fd.io/r/#/c/14744/ > > > > > > > > Other > > jobs have finished more than one and half-hour ago and the patch cannot be > > > > marked > > Verified+1 because Jenkins is still waiting for the ARM job to complete > > > > (timeout > > = 120 minutes). IMHO it makes the overall patch submission and review > > > > very > > painful for authors and commiters. > > > > > > > > I > > would recommend (it has been done in the past for other jobs) to disable > > this > > > > job > > in production, move it again to sandbox, get it fixed and eventually moved > > > > again > > to production... > > > > > > > > - > > Marco > > > > > > > > On > > Fri, 2018-09-07 at 12:04 -0700, Florin Coras wrote: > > > > > ARM jobs have not been working for some days now. That’s why their result > > > is > > > > > > skipped. Timeout is 2h but probably we should drop that even further … > > > > > > > > > > > > Florin > > > > > > > > > > > > > On Sep 7, 2018, at 11:59 AM, Ole Troan <otr...@employees.org> wrote: > > > > > > > > Trying out a change in: > > > > > > > > https://gerrit.fd.io/r/#/c/14732/ > > > > > > > > > > > > > > > > All others succeed but ARM this doesn’t look too good. > > > > > > > > Stuck apparently. > > > > > > > > > > > > > > > > https://jenkins.fd.io/job/vpp-arm-verify-master-ubuntu1604/2157/console > > > > > > > > 20:16:04 > > > > > > > > ======================================================================== > > > > ==== > > > > > > > > == > > > > > > > > > > > > > > > > 20:16:04 > > > > > > > > ARP Test Case > > > > > > > > > > > > > > > > 20:16:04 > > > > > > > > ======================================================================== > > > > ==== > > > > > > > > == > > > > > > > > > > > > > > > > 20:16:07 > > > > > > > > *** Error in `python': free(): invalid pointer: 0x0000ffff553263a8 *** > > > > > > > > > > > > > > > > 20:16:09 > > > > > > > > ARP > > > > OK > > > > > > > > > > > > > > > > 20:16:09 > > > > > > > > ARP Duplicates > > > > OK > > > > > > > > > > > > > > > > 20:16:09 > > > > > > > > test_arp_incomplete (test_neighbor.ARPTestCase) > > > > OK > > > > > > > > > > > > > > > > 20:16:09 > > > > > > > > ARP Static > > > > OK > > > > > > > > > > > > > > > > 20:16:15 > > > > > > > > ARP reply with VRRP virtual src hw addr > > > > OK > > > > > > > > > > > > > > > > 20:16:15 > > > > > > > > GARP > > > > OK > > > > > > > > > > > > > > > > 20:16:15 > > > > > > > > MPLS > > > > OK > > > > > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > Ole > > > > > > > > > > > > > > > > > > > > > > > > > On 7 Sep 2018, at 18:50, Ole Troan <otr...@employees.org> wrote: > > > > > > > > > > > > > > > > > > > > Ed, > > > > > > > > > > > > > > > > > > > > Let me take a closer look at these. > > > > > > > > > > It appears if VPP is slow to start it might not have created the > > > > > socket > > > > > > > > > > yet. Let me try to put in a retry loop and see if that fixes verify. > > > > > > > > > > > > > > > > > > > > Cheers > > > > > > > > > > Ole > > > > > > > > > > > > > > > > > > > > On 7 Sep 2018, at 18:06, Ed Kern via Lists.Fd.Io < > > > > > > > > > > ejk=cisco....@lists.fd.io> wrote: > > > > > > > > > > > > > > > > > > > > > make test failures due to the below causing pretty consistent > > > > > > failures. > > > > > > > > > > > > Note: For whatever reason the failures are not 100%. > > > > > > > > > > > > The failures and with the retries on nonconcurrent merge jobs may > > > > > > lead > > > > > > > > > > > > to long build queues. > > > > > > > > > > > > > > > > > > > > > > > > These are not infra issues but ill be keeping an eye on it. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Ed > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 13:08:25 > > > > > > > > > > > > Using /var/cache/vpp/python/virtualenv/lib/python2.7/site-packages > > > > > > > > > > > > > > > > > > > > > > > > 13:08:25 > > > > > > > > > > > > Finished processing dependencies for vpp-papi==1.6.1 > > > > > > > > > > > > > > > > > > > > > > > > 13:08:27 > > > > > > > > > > > > Traceback (most recent call last): > > > > > > > > > > > > > > > > > > > > > > > > 13:08:27 > > > > > > > > > > > > File "sanity_run_vpp.py", line 21, in <module> > > > > > > > > > > > > > > > > > > > > > > > > 13:08:27 > > > > > > > > > > > > tc.setUpClass() > > > > > > > > > > > > > > > > > > > > > > > > 13:08:27 > > > > > > > > > > > > File "/w/workspace/vpp-merge-master-ubuntu1604/test/framework.py", > > > > > > line > > > > > > > > > > > > 394, in setUpClass > > > > > > > > > > > > > > > > > > > > > > > > 13:08:27 > > > > > > > > > > > > cls.statistics = VPPStats(socketname=cls.tempdir+'/stats.sock') > > > > > > > > > > > > > > > > > > > > > > > > 13:08:27 > > > > > > > > > > > > File "build/bdist.linux-x86_64/egg/vpp_papi/vpp_stats.py", line 117, > > > > > > in > > > > > > > > > > > > __init__ > > > > > > > > > > > > > > > > > > > > > > > > 13:08:27 > > > > > > > > > > > > IOError > > > > > > > > > > > > > > > > > > > > > > > > 13:08:27 > > > > > > > > > > > > ******************************************************************* > > > > > > > > > > > > > > > > > > > > > > > > 13:08:27 > > > > > > > > > > > > * Sanity check failed, cannot run vpp > > > > > > > > > > > > > > > > > > > > > > > > 13:08:27 > > > > > > > > > > > > ******************************************************************* > > > > > > > > > > > > > > > > > > > > > > > > -=-=-=-=-=-=-=-=-=-=-=- > > > > > > > > > > > > Links: You receive all messages sent to this group. > > > > > > > > > > > > > > > > > > > > > > > > View/Reply Online (#10432): > > > > > > https://lists.fd.io/g/vpp-dev/message/10432 > > > > > > > > > > > > Mute This Topic: https://lists.fd.io/mt/25308161/675193 > > > > > > > > > > > > Group Owner: vpp-dev+ow...@lists.fd.io > > > > > > > > > > > > Unsubscribe: https://lists.fd.io/g/vpp-dev/unsubb [otroan@employees > > > > > > .org > > > > > > > > > > > > ] > > > > > > > > > > > > -=-=-=-=-=-=-=-=-=-=-=- > > > > > > > > > > > > > > > > -=-=-=-=-=-=-=-=-=-=-=- > > > > > > > > > > Links: You receive all messages sent to this group. > > > > > > > > > > > > > > > > > > > > View/Reply Online (#10435): > > > > > https://lists.fd.io/g/vpp-dev/message/10435 > > > > > > > > > > Mute This Topic: https://lists.fd.io/mt/25308161/675193 > > > > > > > > > > Group Owner: vpp-dev+ow...@lists.fd.io > > > > > > > > > > Unsubscribe: https://lists.fd.io/g/vpp-dev/unsubbb [otroan@employees. > > > > > org] > > > > > > > > > > -=-=-=-=-=-=-=-=-=-=-=- > > > > > > > > > > > > > > > > > -=-=-=-=-=-=-=-=-=-=-=- > > > > > > > > Links: You receive all messages sent to this group. > > > > > > > > > > > > > > > > View/Reply Online (#10437): https://lists.fd.io/g/vpp-dev/message/10437 > > > > > > > > Mute This Topic: https://lists.fd.io/mt/25308161/675152 > > > > > > > > Group Owner: vpp-dev+ow...@lists.fd.io > > > > > > > > Unsubscribe: https://lists.fd.io/g/vpp-dev/unsubbb [fcoras.lists@gmail. > > > > com] > > > > > > > > -=-=-=-=-=-=-=-=-=-=-=- > > > > > > > > > > > > > -=-=-=-=-=-=-=-=-=-=-=- > > > > > > Links: You receive all messages sent to this group. > > > > > > > > > > > > View/Reply Online (#10438): https://lists.fd.io/g/vpp-dev/message/10438 > > > > > > Mute This Topic: https://lists.fd.io/mt/25308161/675056 > > > > > > Group Owner: vpp-dev+ow...@lists.fd.io > > > > > > Unsubscribe: https://lists.fd.io/g/vpp-dev/unsubbb [mvarl...@suse.de] > > > > > > -=-=-=-=-=-=-=-=-=-=-=- > > > > > > > > > > > > > -- > > > > Marco > > V > > > > > > > > SUSE > > LINUX GmbH | GF: Felix Imendörffer, Jane Smithard, Graham Norton > > > > HRB > > 21284 (AG Nürnberg) Maxfeldstr. 5, D-90409, Nürnberg > > > > > > > > -=-=-=-=-=-=-=-=-=-=-=- > > > > Links: > > You receive all messages sent to this group. > > > > > > > > View/Reply > > Online (#10453): https://lists.fd.io/g/vpp-dev/message/10453 > > > > Mute > > This Topic: https://lists.fd.io/mt/25308161/675649 > > > > Group > > Owner: vpp-dev+ow...@lists.fd.io > > > > Unsubscribe: https://lists.fd.io/g/vpp-dev/unsubb [e...@cisco.com] > > > > -=-=-=-=-=-=-=-=-=-=-=- > > > > > > > -- Marco V
SUSE LINUX GmbH | GF: Felix Imendörffer, Jane Smithard, Graham Norton HRB 21284 (AG Nürnberg) Maxfeldstr. 5, D-90409, Nürnberg
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#10460): https://lists.fd.io/g/vpp-dev/message/10460 Mute This Topic: https://lists.fd.io/mt/25308161/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
