Thanks, Dave!
I’ll take a look at those as soon as I can. I’m running multiple
connections between 2 vpp hosts without issue, so it’s either a
cut-through session issue or it has to do with how we setup vpp for vrf
leaking.
Cheers,
Florin
On Nov 15, 2018, at 3:00 PM, Dave Wallace <[1]dwallac...@gmail.com>
wrote:
Same here. However, in the same workspace where all tests passed, I can
get this test case to fail consistently:
EXTENDED_TESTS=y TEST=vcl.VCLThruHostStackExtendedBTestCase.* make test
EXTENDED_TESTS=y TEST=vcl.VCLIpv6ThruHostStackExtendedBTestCase.* make
test
In patch 13215, I discovered that making these test cases NOT run
multiple sockets in parallel the test passes. My latest patch to that
has the multiple sockets option commented out with "# ouch! Host Stack
Bug?" so that all tests pass.
Thanks,
-daw-
On 11/15/2018 4:16 PM, Florin Coras wrote:
That’s an interesting failure. Is the test machine running out of memory?
The extended tests are unstable on my server, so I do see quite a number of
failures. However this:
make test-ext TEST=vcl.VCLCutThruTestCase.test_vcl_cut_thru_uni_dir_nsock
runs just fine. After the latest test framework changes, are we running
multiple tests/vpps in parallel? I suspect that may be a source of issues.
Florin
On Nov 15, 2018, at 12:11 PM, Klement Sekera via Lists.Fd.Io
[2]<ksekera=cisco....@lists.fd.io> wrote:
I'm seeing timeouts and coredumps...
e.g.
#6 0x00007f9ba0404eb6 in svm_msg_q_try_lock (mq=0x204009440)
at /home/ksekera/vpp/src/svm/message_queue.h:299
299 return pthread_mutex_trylock (&mq->q->mutex);
(gdb) p mq
$1 = (svm_msg_q_t *) 0x204009440
(gdb) p mq->q
$2 = (svm_queue_t *) 0x0
which is part of
#4 <signal handler called>
#5 __pthread_mutex_trylock (mutex=0x0) at ../nptl/pthread_mutex_trylock.c:39
#6 0x00007f9ba0404eb6 in svm_msg_q_try_lock (mq=0x204009440)
at /home/ksekera/vpp/src/svm/message_queue.h:299
#7 0x00007f9ba04055d5 in svm_msg_q_lock_and_alloc_msg_w_ring (mq=0x204009440,
ring_index=1, noblock=1 '\001', msg=0x7f9b5f7c2a80)
at /home/ksekera/vpp/src/svm/message_queue.c:121
#8 0x00007f9ba14be449 in mq_try_lock_and_alloc_msg (app_mq=0x204009440,
msg=0x7f9b5f7c2a80) at /home/ksekera/vpp/src/vnet/session/session_api.c:407
#9 0x00007f9ba14be509 in mq_send_session_accepted_cb (s=0x7f9b60351400)
at /home/ksekera/vpp/src/vnet/session/session_api.c:432
#10 0x00007f9ba1496ba0 in application_local_session_connect (
client_wrk=0x7f9b60805800, server_wrk=0x7f9b60805780, ll=0x7f9b5f4c9e40,
opaque=0) at /home/ksekera/vpp/src/vnet/session/application.c:1646
#11 0x00007f9ba14a5a62 in application_connect (a=0x7f9b5f7c2d30)
at /home/ksekera/vpp/src/vnet/session/application_interface.c:327
---Type <return> to continue, or q <return> to quit---
#12 0x00007f9ba14a69fd in vnet_connect (a=0x7f9b5f7c2d30)
at /home/ksekera/vpp/src/vnet/session/application_interface.c:673
#13 0x00007f9ba14c0f27 in vl_api_connect_sock_t_handler (mp=0x1300a6218)
at /home/ksekera/vpp/src/vnet/session/session_api.c:1305
#14 0x00007f9ba1b6cb25 in vl_msg_api_handler_with_vm_node (
am=0x7f9ba1d7dc60 <api_main>, the_msg=0x1300a6218,
vm=0x7f9ba08fc2c0 <vlib_global_main>, node=0x7f9b5f7ba000)
at /home/ksekera/vpp/src/vlibapi/api_shared.c:502
#15 0x00007f9ba1b39114 in void_mem_api_handle_msg_i (
am=0x7f9ba1d7dc60 <api_main>, vm=0x7f9ba08fc2c0 <vlib_global_main>,
node=0x7f9b5f7ba000, q=0x13004c440)
at /home/ksekera/vpp/src/vlibmemory/memory_api.c:700
#16 0x00007f9ba1b39183 in vl_mem_api_handle_msg_main (
vm=0x7f9ba08fc2c0 <vlib_global_main>, node=0x7f9b5f7ba000)
at /home/ksekera/vpp/src/vlibmemory/memory_api.c:710
#17 0x00007f9ba1b572dd in vl_api_clnt_process (
vm=0x7f9ba08fc2c0 <vlib_global_main>, node=0x7f9b5f7ba000, f=0x0)
at /home/ksekera/vpp/src/vlibmemory/vlib_api.c:350
#18 0x00007f9ba0674a11 in vlib_process_bootstrap (_a=140305300978672)
at /home/ksekera/vpp/src/vlib/main.c:1276
#19 0x00007f9b9fef4e74 in clib_calljmp ()
from
/home/ksekera/vpp/build-root/install-vpp_debug-native/vpp/lib/libvppinfra.so.19.01
could this be the result of a timeout and the killing of the child
process?
Thanks,
Klement
Quoting Dave Wallace (2018-11-15 20:27:55)
Klement,
I just pulled the top-of-tree on master and ran only VCL tests on my 18.04
box and they all passed (see below). Another strange thing about your
failure is that the test that failed is NOT an extended test.
I'm currently working on a patch ([1][3]https://gerrit.fd.io/r/#/c/13215/) to
shorten the run time for the extended tests and convert them to regular
tests. In the past, I have seen some unexplained failures of some of the
extended tests. I'll let you know if I encounter any of them again.
Thanks,
-daw-
----- %< -----
TEST=vcl.* make test-ext
. . .
make[2]: Leaving directory '/scratch/dwallacelf/lf/vpp/test/ext'
==============================================================================
Sanity test case - verify if VPP is able to start
==============================================================================
Running tests using custom test runner
Active filters: file=test_vcl.py, class=None, function=None
Adding tests from directory tree /scratch/dwallacelf/lf/vpp/test
28 out of 858 tests match specified filters
==============================================================================
VCL Thru Host Stack Tests
==============================================================================
run LDP thru host stack echo test
OK
run VCL thru host stack echo test
OK
==============================================================================
VCL Thru Host Stack Extended B Tests
==============================================================================
run LDP thru host stack bi-directional (multiple sockets) test
OK
==============================================================================
VCL Thru Host Stack Iperf Tests
==============================================================================
run LDP thru host stack iperf3 test
OK
==============================================================================
VCL Thru Host Stack Extended D Tests
==============================================================================
run VCL thru host stack uni-directional (multiple sockets) test
OK
==============================================================================
VCL IPv6 Thru Host Stack Tests
==============================================================================
run LDP IPv6 thru host stack echo test
OK
run VCL IPv6 thru host stack echo test
OK
==============================================================================
VCL Cut Thru Tests
==============================================================================
run LDP cut thru bi-directional (multiple sockets) test
OK
run LDP cut thru echo test
OK
run LDP cut thru iperf3 test
OK
run LDP cut thru uni-directional (multiple sockets) test
OK
run VCL cut thru bi-directional (multiple sockets) test
OK
run VCL cut thru echo test
OK
run VCL cut thru uni-directional (multiple sockets) test
OK
==============================================================================
VCL Thru Host Stack Extended C Tests
==============================================================================
run LDP thru host stack uni-directional (multiple sockets) test
OK
==============================================================================
VCL Thru Host Stack NSession Bidir Tests
==============================================================================
run VCL thru host stack bi-directional (multiple sockets) test
OK
==============================================================================
VCL IPv6 Thru Host Stack Extended A Tests
==============================================================================
run VCL thru host stack bi-directional (multiple sockets) test
OK
==============================================================================
VCL IPv6 Cut Thru Tests
==============================================================================
run LDP IPv6 cut thru bi-directional (multiple sockets) test
OK
run LDP IPv6 cut thru echo test
OK
run LDP IPv6 cut thru iperf3 test
OK
run LDP IPv6 cut thru uni-directional (multiple sockets) test
OK
run VCL IPv6 cut thru bi-directional (multiple sockets) test
OK
run VCL IPv6 cut thru echo test
OK
run VCL IPv6 cut thru uni-directional (multiple sockets) test
OK
==============================================================================
VCL IPv6 Thru Host Stack Extended C Tests
==============================================================================
run LDP thru host stack uni-directional (multiple sockets) test
OK
==============================================================================
VCL IPv6 Thru Host Stack Extended B Tests
==============================================================================
run LDP thru host stack bi-directional (multiple sockets) test
OK
==============================================================================
VCL IPv6 Thru Host Stack Iperf Tests
==============================================================================
run LDP thru host stack iperf3 test
OK
==============================================================================
VCL IPv6 Thru Host Stack Extended D Tests
==============================================================================
run VCL thru host stack uni-directional (multiple sockets) test
OK
Ran 28 tests in 250.800s
OK
==============================================================================
TEST RESULTS:
Scheduled tests: 28
Executed tests: 28
Passed tests: 28
==============================================================================
Test run was successful
----- %< -----
On 11/15/2018 5:47 AM, Klement Sekera via Lists.Fd.Io wrote:
Hi all,
I'm seeing failures on master branch on ubuntu 18.04 when invoking `make
test-ext`
FAILURES AND ERRORS IN TESTS:
Testcase name: VCL Cut Thru Tests
FAILURE: run VCL cut thru uni-directional (multiple sockets) test
Testcase name: L2BD Test Case
ERROR: L2BD MAC learning dual-loop test
ERROR: L2BD MAC learning dual-loop test
ERROR: L2BD MAC learning single-loop test
Testcase name: Classifier PBR Test Case
ERROR: IP PBR test
digging a bit further, L2BD failure also occurs in `make test-debug`,
while it doesn't appear in `make test`. This is a core due to assert.
I think we should run both `make test` (release build) and `make
test-debug` (debug build) as part of verify process. If it was up to me,
I would run all the tests which we have in the verify job.
Thoughts?
Regards,
Klement
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#11257): [2][4]https://lists.fd.io/g/vpp-dev/message/11257
Mute This Topic: [3][5]https://lists.fd.io/mt/28144643/675079
Group Owner: [[6]4]vpp-dev+ow...@lists.fd.io
Unsubscribe: [5][7]https://lists.fd.io/g/vpp-dev/unsub
[[[8]6]dwallac...@gmail.com]
-=-=-=-=-=-=-=-=-=-=-=-
References
Visible links
1. [9]https://gerrit.fd.io/r/#/c/13215/
2. [10]https://lists.fd.io/g/vpp-dev/message/11257
3. [11]https://lists.fd.io/mt/28144643/675079
4. [12]mailto:vpp-dev+ow...@lists.fd.io
5. [13]https://lists.fd.io/g/vpp-dev/unsub
6. [14]mailto:dwallac...@gmail.com
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#11270): [15]https://lists.fd.io/g/vpp-dev/message/11270
Mute This Topic: [16]https://lists.fd.io/mt/28144643/675152
Group Owner: [17]vpp-dev+ow...@lists.fd.io
Unsubscribe: [18]https://lists.fd.io/g/vpp-dev/unsub
[[19]fcoras.li...@gmail.com]
-=-=-=-=-=-=-=-=-=-=-=-
References
Visible links
1. mailto:dwallac...@gmail.com
2. mailto:ksekera=cisco....@lists.fd.io
3. https://gerrit.fd.io/r/#/c/13215/
4. https://lists.fd.io/g/vpp-dev/message/11257
5. https://lists.fd.io/mt/28144643/675079
6. mailto:4]vpp-dev+ow...@lists.fd.io
7. https://lists.fd.io/g/vpp-dev/unsub
8. mailto:6]dwallac...@gmail.com
9. https://gerrit.fd.io/r/#/c/13215/
10. https://lists.fd.io/g/vpp-dev/message/11257
11. https://lists.fd.io/mt/28144643/675079
12. mailto:vpp-dev+ow...@lists.fd.io
13. https://lists.fd.io/g/vpp-dev/unsub
14. mailto:dwallac...@gmail.com
15. https://lists.fd.io/g/vpp-dev/message/11270
16. https://lists.fd.io/mt/28144643/675152
17. mailto:vpp-dev+ow...@lists.fd.io
18. https://lists.fd.io/g/vpp-dev/unsub
19. mailto:fcoras.li...@gmail.com
