There is also an attempt to clean up processes left bind in a shell script test/scripts/run_in_venv_with_cleanup.sh . I don't understand why it doesn't work in this case...
Thanks, Klement Quoting Dave Wallace (2018-11-16 20:39:23) > I'll see if I can reproduce the issue. > > I seem to recall some python test code which was supposed to detect that > condition and clean up, but I don't think that I ever verified that it > worked. > > Thanks, > -daw- > On 11/16/2018 11:44 AM, Florin Coras wrote: > > I also noticed that although it shouldn’t be different from the vcl test > apps which seem to be properly killed on test failure. Anyway, it’s on my > todo list, if nobody beats me to it. > > Florin > > > On Nov 16, 2018, at 5:27 AM, Klement Sekera [1]<ksek...@cisco.com> wrote: > > I've also noticed that quite often a binary called sock_test_client is > left running after a vpp crash or test failure. What's worse, it eats > 100% CPU. > > Quoting Florin Coras (2018-11-16 00:40:06) > > Thanks, Dave! > I’ll take a look at those as soon as I can. I’m running multiple > connections between 2 vpp hosts without issue, so it’s either a > cut-through session issue or it has to do with how we setup vpp for vrf > leaking. > Cheers, > Florin > > On Nov 15, 2018, at 3:00 PM, Dave Wallace [2]<[1]dwallac...@gmail.com> > wrote: > Same here. However, in the same workspace where all tests passed, I can > get this test case to fail consistently: > > EXTENDED_TESTS=y TEST=vcl.VCLThruHostStackExtendedBTestCase.* make test > EXTENDED_TESTS=y TEST=vcl.VCLIpv6ThruHostStackExtendedBTestCase.* make > test > > In patch 13215, I discovered that making these test cases NOT run > multiple sockets in parallel the test passes. My latest patch to that > has the multiple sockets option commented out with "# ouch! Host Stack > Bug?" so that all tests pass. > > Thanks, > -daw- > On 11/15/2018 4:16 PM, Florin Coras wrote: > > That’s an interesting failure. Is the test machine running out of memory? > > The extended tests are unstable on my server, so I do see quite a number of > failures. However this: > > make test-ext TEST=vcl.VCLCutThruTestCase.test_vcl_cut_thru_uni_dir_nsock > > runs just fine. After the latest test framework changes, are we running > multiple tests/vpps in parallel? I suspect that may be a source of issues. > > Florin > > > On Nov 15, 2018, at 12:11 PM, Klement Sekera via Lists.Fd.Io > [2][3]<ksekera=cisco....@lists.fd.io> wrote: > > I'm seeing timeouts and coredumps... > > e.g. > > #6 0x00007f9ba0404eb6 in svm_msg_q_try_lock (mq=0x204009440) > at /home/ksekera/vpp/src/svm/message_queue.h:299 > 299 return pthread_mutex_trylock (&mq->q->mutex); > (gdb) p mq > $1 = (svm_msg_q_t *) 0x204009440 > (gdb) p mq->q > $2 = (svm_queue_t *) 0x0 > > which is part of > > #4 <signal handler called> > #5 __pthread_mutex_trylock (mutex=0x0) at ../nptl/pthread_mutex_trylock.c:39 > #6 0x00007f9ba0404eb6 in svm_msg_q_try_lock (mq=0x204009440) > at /home/ksekera/vpp/src/svm/message_queue.h:299 > #7 0x00007f9ba04055d5 in svm_msg_q_lock_and_alloc_msg_w_ring > (mq=0x204009440, > ring_index=1, noblock=1 '\001', msg=0x7f9b5f7c2a80) > at /home/ksekera/vpp/src/svm/message_queue.c:121 > #8 0x00007f9ba14be449 in mq_try_lock_and_alloc_msg (app_mq=0x204009440, > msg=0x7f9b5f7c2a80) at > /home/ksekera/vpp/src/vnet/session/session_api.c:407 > #9 0x00007f9ba14be509 in mq_send_session_accepted_cb (s=0x7f9b60351400) > at /home/ksekera/vpp/src/vnet/session/session_api.c:432 > #10 0x00007f9ba1496ba0 in application_local_session_connect ( > client_wrk=0x7f9b60805800, server_wrk=0x7f9b60805780, ll=0x7f9b5f4c9e40, > opaque=0) at /home/ksekera/vpp/src/vnet/session/application.c:1646 > #11 0x00007f9ba14a5a62 in application_connect (a=0x7f9b5f7c2d30) > at /home/ksekera/vpp/src/vnet/session/application_interface.c:327 > ---Type <return> to continue, or q <return> to quit--- > #12 0x00007f9ba14a69fd in vnet_connect (a=0x7f9b5f7c2d30) > at /home/ksekera/vpp/src/vnet/session/application_interface.c:673 > #13 0x00007f9ba14c0f27 in vl_api_connect_sock_t_handler (mp=0x1300a6218) > at /home/ksekera/vpp/src/vnet/session/session_api.c:1305 > #14 0x00007f9ba1b6cb25 in vl_msg_api_handler_with_vm_node ( > am=0x7f9ba1d7dc60 <api_main>, the_msg=0x1300a6218, > vm=0x7f9ba08fc2c0 <vlib_global_main>, node=0x7f9b5f7ba000) > at /home/ksekera/vpp/src/vlibapi/api_shared.c:502 > #15 0x00007f9ba1b39114 in void_mem_api_handle_msg_i ( > am=0x7f9ba1d7dc60 <api_main>, vm=0x7f9ba08fc2c0 <vlib_global_main>, > node=0x7f9b5f7ba000, q=0x13004c440) > at /home/ksekera/vpp/src/vlibmemory/memory_api.c:700 > #16 0x00007f9ba1b39183 in vl_mem_api_handle_msg_main ( > vm=0x7f9ba08fc2c0 <vlib_global_main>, node=0x7f9b5f7ba000) > at /home/ksekera/vpp/src/vlibmemory/memory_api.c:710 > #17 0x00007f9ba1b572dd in vl_api_clnt_process ( > vm=0x7f9ba08fc2c0 <vlib_global_main>, node=0x7f9b5f7ba000, f=0x0) > at /home/ksekera/vpp/src/vlibmemory/vlib_api.c:350 > #18 0x00007f9ba0674a11 in vlib_process_bootstrap (_a=140305300978672) > at /home/ksekera/vpp/src/vlib/main.c:1276 > #19 0x00007f9b9fef4e74 in clib_calljmp () > from > /home/ksekera/vpp/build-root/install-vpp_debug-native/vpp/lib/libvppinfra.so.19.01 > > could this be the result of a timeout and the killing of the child > process? > > Thanks, > Klement > > > Quoting Dave Wallace (2018-11-15 20:27:55) > > Klement, > > I just pulled the top-of-tree on master and ran only VCL tests on my 18.04 > box and they all passed (see below). Another strange thing about your > failure is that the test that failed is NOT an extended test. > > I'm currently working on a patch > ([1][3][4]https://gerrit.fd.io/r/#/c/13215/) to > shorten the run time for the extended tests and convert them to regular > tests. In the past, I have seen some unexplained failures of some of the > extended tests. I'll let you know if I encounter any of them again. > > Thanks, > -daw- > > ----- %< ----- > TEST=vcl.* make test-ext > . . . > make[2]: Leaving directory '/scratch/dwallacelf/lf/vpp/test/ext' > > ============================================================================== > Sanity test case - verify if VPP is able to start > > ============================================================================== > Running tests using custom test runner > Active filters: file=test_vcl.py, class=None, function=None > Adding tests from directory tree /scratch/dwallacelf/lf/vpp/test > 28 out of 858 tests match specified filters > > ============================================================================== > VCL Thru Host Stack Tests > > ============================================================================== > run LDP thru host stack echo test > OK > run VCL thru host stack echo test > OK > > ============================================================================== > VCL Thru Host Stack Extended B Tests > > ============================================================================== > run LDP thru host stack bi-directional (multiple sockets) test > OK > > ============================================================================== > VCL Thru Host Stack Iperf Tests > > ============================================================================== > run LDP thru host stack iperf3 test > OK > > ============================================================================== > VCL Thru Host Stack Extended D Tests > > ============================================================================== > run VCL thru host stack uni-directional (multiple sockets) test > OK > > ============================================================================== > VCL IPv6 Thru Host Stack Tests > > ============================================================================== > run LDP IPv6 thru host stack echo test > OK > run VCL IPv6 thru host stack echo test > OK > > ============================================================================== > VCL Cut Thru Tests > > ============================================================================== > run LDP cut thru bi-directional (multiple sockets) test > OK > run LDP cut thru echo test > OK > run LDP cut thru iperf3 test > OK > run LDP cut thru uni-directional (multiple sockets) test > OK > run VCL cut thru bi-directional (multiple sockets) test > OK > run VCL cut thru echo test > OK > run VCL cut thru uni-directional (multiple sockets) test > OK > > ============================================================================== > VCL Thru Host Stack Extended C Tests > > ============================================================================== > run LDP thru host stack uni-directional (multiple sockets) test > OK > > ============================================================================== > VCL Thru Host Stack NSession Bidir Tests > > ============================================================================== > run VCL thru host stack bi-directional (multiple sockets) test > OK > > ============================================================================== > VCL IPv6 Thru Host Stack Extended A Tests > > ============================================================================== > run VCL thru host stack bi-directional (multiple sockets) test > OK > > ============================================================================== > VCL IPv6 Cut Thru Tests > > ============================================================================== > run LDP IPv6 cut thru bi-directional (multiple sockets) test > OK > run LDP IPv6 cut thru echo test > OK > run LDP IPv6 cut thru iperf3 test > OK > run LDP IPv6 cut thru uni-directional (multiple sockets) test > OK > run VCL IPv6 cut thru bi-directional (multiple sockets) test > OK > run VCL IPv6 cut thru echo test > OK > run VCL IPv6 cut thru uni-directional (multiple sockets) test > OK > > ============================================================================== > VCL IPv6 Thru Host Stack Extended C Tests > > ============================================================================== > run LDP thru host stack uni-directional (multiple sockets) test > OK > > ============================================================================== > VCL IPv6 Thru Host Stack Extended B Tests > > ============================================================================== > run LDP thru host stack bi-directional (multiple sockets) test > OK > > ============================================================================== > VCL IPv6 Thru Host Stack Iperf Tests > > ============================================================================== > run LDP thru host stack iperf3 test > OK > > ============================================================================== > VCL IPv6 Thru Host Stack Extended D Tests > > ============================================================================== > run VCL thru host stack uni-directional (multiple sockets) test > OK > > Ran 28 tests in 250.800s > > OK > > > ============================================================================== > TEST RESULTS: > Scheduled tests: 28 > Executed tests: 28 > Passed tests: 28 > > ============================================================================== > > Test run was successful > ----- %< ----- > On 11/15/2018 5:47 AM, Klement Sekera via Lists.Fd.Io wrote: > > Hi all, > > I'm seeing failures on master branch on ubuntu 18.04 when invoking `make > test-ext` > > FAILURES AND ERRORS IN TESTS: > Testcase name: VCL Cut Thru Tests > FAILURE: run VCL cut thru uni-directional (multiple sockets) test > Testcase name: L2BD Test Case > ERROR: L2BD MAC learning dual-loop test > ERROR: L2BD MAC learning dual-loop test > ERROR: L2BD MAC learning single-loop test > Testcase name: Classifier PBR Test Case > ERROR: IP PBR test > > digging a bit further, L2BD failure also occurs in `make test-debug`, > while it doesn't appear in `make test`. This is a core due to assert. > > I think we should run both `make test` (release build) and `make > test-debug` (debug build) as part of verify process. If it was up to me, > I would run all the tests which we have in the verify job. > > Thoughts? > > Regards, > Klement > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > > View/Reply Online (#11257): > [2][4][5]https://lists.fd.io/g/vpp-dev/message/11257 > Mute This Topic: [3][5][6]https://lists.fd.io/mt/28144643/675079 > Group Owner: [[[7]6]4]vpp-dev+ow...@lists.fd.io > Unsubscribe: [5][7][8]https://lists.fd.io/g/vpp-dev/unsub > [[[[9]8]6]dwallac...@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- > > References > > Visible links > 1. [9][10]https://gerrit.fd.io/r/#/c/13215/ > 2. [10][11]https://lists.fd.io/g/vpp-dev/message/11257 > 3. [11][12]https://lists.fd.io/mt/28144643/675079 > 4. [12][13]mailto:vpp-dev+ow...@lists.fd.io > 5. [13][14]https://lists.fd.io/g/vpp-dev/unsub > 6. [14][15]mailto:dwallac...@gmail.com > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > > View/Reply Online (#11270): > [15][16]https://lists.fd.io/g/vpp-dev/message/11270 > Mute This Topic: [16][17]https://lists.fd.io/mt/28144643/675152 > Group Owner: [[18]17]vpp-dev+ow...@lists.fd.io > Unsubscribe: [18][19]https://lists.fd.io/g/vpp-dev/unsub > [[[20]19]fcoras.li...@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- > > References > > Visible links > 1. [21]mailto:dwallac...@gmail.com > 2. [22]mailto:ksekera=cisco....@lists.fd.io > 3. [23]https://gerrit.fd.io/r/#/c/13215/ > 4. [24]https://lists.fd.io/g/vpp-dev/message/11257 > 5. [25]https://lists.fd.io/mt/28144643/675079 > 6. [26]mail[27]mailto:4]vpp-dev+ow...@lists.fd.io > 7. [28]https://lists.fd.io/g/vpp-dev/unsub > 8. [29]mail[30]mailto:6]dwallac...@gmail.com > 9. [31]https://gerrit.fd.io/r/#/c/13215/ > 10. [32]https://lists.fd.io/g/vpp-dev/message/11257 > 11. [33]https://lists.fd.io/mt/28144643/675079 > 12. [34]mailto:vpp-dev+ow...@lists.fd.io > 13. [35]https://lists.fd.io/g/vpp-dev/unsub > 14. [36]mailto:dwallac...@gmail.com > 15. [37]https://lists.fd.io/g/vpp-dev/message/11270 > 16. [38]https://lists.fd.io/mt/28144643/675152 > 17. [39]mailto:vpp-dev+ow...@lists.fd.io > 18. [40]https://lists.fd.io/g/vpp-dev/unsub > 19. [41]mailto:fcoras.li...@gmail.com > > References > > Visible links > 1. mailto:ksek...@cisco.com > 2. mailto:[1]dwallac...@gmail.com > 3. mailto:ksekera=cisco....@lists.fd.io > 4. https://gerrit.fd.io/r/#/c/13215/ > 5. https://lists.fd.io/g/vpp-dev/message/11257 > 6. https://lists.fd.io/mt/28144643/675079 > 7. mailto:6]4]vpp-dev+ow...@lists.fd.io > 8. https://lists.fd.io/g/vpp-dev/unsub > 9. mailto:8]6]dwallac...@gmail.com > 10. https://gerrit.fd.io/r/#/c/13215/ > 11. https://lists.fd.io/g/vpp-dev/message/11257 > 12. https://lists.fd.io/mt/28144643/675079 > 13. mailto:vpp-dev+ow...@lists.fd.io > 14. https://lists.fd.io/g/vpp-dev/unsub > 15. mailto:dwallac...@gmail.com > 16. https://lists.fd.io/g/vpp-dev/message/11270 > 17. https://lists.fd.io/mt/28144643/675152 > 18. mailto:17]vpp-dev+ow...@lists.fd.io > 19. https://lists.fd.io/g/vpp-dev/unsub > 20. mailto:19]fcoras.li...@gmail.com > 21. mailto:dwallac...@gmail.com > 22. mailto:ksekera=cisco....@lists.fd.io > 23. https://gerrit.fd.io/r/#/c/13215/ > 24. https://lists.fd.io/g/vpp-dev/message/11257 > 25. https://lists.fd.io/mt/28144643/675079 > 26. mailto:4 > 27. mailto:mailto:4]vpp-dev+ow...@lists.fd.io > 28. https://lists.fd.io/g/vpp-dev/unsub > 29. mailto:6 > 30. mailto:mailto:6]dwallac...@gmail.com > 31. https://gerrit.fd.io/r/#/c/13215/ > 32. https://lists.fd.io/g/vpp-dev/message/11257 > 33. https://lists.fd.io/mt/28144643/675079 > 34. mailto:vpp-dev+ow...@lists.fd.io > 35. https://lists.fd.io/g/vpp-dev/unsub > 36. mailto:dwallac...@gmail.com > 37. https://lists.fd.io/g/vpp-dev/message/11270 > 38. https://lists.fd.io/mt/28144643/675152 > 39. mailto:vpp-dev+ow...@lists.fd.io > 40. https://lists.fd.io/g/vpp-dev/unsub > 41. mailto:fcoras.li...@gmail.com
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#11308): https://lists.fd.io/g/vpp-dev/message/11308 Mute This Topic: https://lists.fd.io/mt/28144643/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
