No need for such crude approach ;-)
ksekera@zglab-host-4 ~/vpp> make test-shell
...
(virtualenv) ksekera@zglab-host-4:~/vpp/test$ ./discover_tests.py
...
test_jvpp.py.TestJVpp.test_vpp_snat_callback_api
test_jvpp.py.TestJVpp.test_vpp_snat_future_api
test_dhcp.py.TestDHCP.test_dhcp6_proxy
test_dhcp.py.TestDHCP.test_dhcp_client
test_dhcp.py.TestDHCP.test_dhcp_proxy
(virtualenv) ksekera@zglab-host-4:~/vpp/test$
by changing "print_callback" in discover_tests.py, one can customize the
format or the action to be done for each test case (default being printing
the test as show above).
Regards,
Klement
Quoting Dave Wallace (2017-10-02 17:31:28)
> Brian,
>
> A brute-force means of achieving your goal would be to use the "TEST="
> option to run each test individually:
>
> TEST=<filter> - filter the set of tests:
> by file-name - only run tests from specified file, e.g.
> TEST=test_bfd selects all tests from test_bfd.py
> by file-suffix - same as file-name, but 'test_' is omitted e.g.
> TEST=bfd selects all tests from test_bfd.py
> by wildcard - wildcard filter is <file>.<class>.<test function>,
> each can be replaced by '*'
> e.g. TEST='test_bfd.*.*' is equivalent to above
> example of filter by file-name
> TEST='bfd.*.*' is equivalent to above example
> of filter by file-suffix
> TEST='bfd.BFDAPITestCase.*' selects all tests
> from test_bfd.py which are part of BFDAPITestCase class
> TEST='bfd.BFDAPITestCase.test_add_bfd'
> selects a single test named test_add_bfd from test_bfd.py/BFDAPITestCase
> TEST='*.*.test_add_bfd' selects all test
> functions named test_add_bfd from all files/classes
>
> With a bit of bash scripting, this turns out to be quite easy. Here's a
> one-liner which executes all of the l2xc tests in
> .../vpp/test/test_l2xc.py:
>
> for test in $(grep "def test_" test/test_l2xc.py | awk -e '{print $2}' |
> cut -d'(' -f1) ; do make TEST="*.*.$test" test ; done
>
> Thanks,
> -daw-
> On 10/02/2017 03:44 AM, Klement Sekera -X (ksekera - PANTHEON TECHNOLOGIES
> at Cisco) wrote:
>
> Hi Brian,
>
> FAILFAST=0 is the default, so there is no need to specify it.
>
> Usually, when a test case 'runs forever', it means that the VPP crashed
> while processing an API call (did you see a message about a core found
> in temporary directory?), in which case python will be stuck waiting for
> a lock. Now, the problem is that there is no way to cancel that stuck
> thread in Python (which is for most cases the main thread).
>
> So to work around this issue, the following approach is taken: First
> thing the test framework does is fork - child continues running
> the tests while parent monitors the child. There is a pipe open,
> through which child sends keep-alive objects containing the name of
> the test case, its temporary directory and some other info whenever
> setUp() runs. That way parent can tell that the child is progressing
> through the tests and also can say which test caused timeout.
>
> I spent quite some time trying to make it work so that the tests can
> continue, but unless we refactor the code so that each test runs in its
> own forked process, we won't be able to achieve your goal. So far, 100%
> of these timeouts were caused by VPP dumping a core and it seemed to me
> like a waste of time spending extra time to make the tests finish.
>
> Thanks,
> Klement
>
> Quoting Brian Brooks (2017-09-22 19:07:53)
>
> Is there a way to make:
>
> FAILFAST=0 TIMEOUT=t make test
>
> actually continue on to the next test if a test has reached timeout t?
>
> The motivation is that I see at least one test case running for forever,
> and I want to be able to see how many test cases are failing in total.
>
> I didn't see that this was possible to do out of the box, so I tried one
> approach: launch a thread that waits until the timeout has passed and then
> send a term signal to the vpp process. If the test case completes before
> timeout, the test case's "tearDown" routine will cancel the timeout thread.
> The timeout and killing part works, except the test framework still does
> not continue on to the next test case.
>
> Thoughts?
>
> Thanks,
> Brian
> _______________________________________________
> vpp-dev mailing list
> [1]vpp-dev@lists.fd.io
> [2]https://lists.fd.io/mailman/listinfo/vpp-dev
>
> _______________________________________________
> vpp-dev mailing list
> [3]vpp-dev@lists.fd.io
> [4]https://lists.fd.io/mailman/listinfo/vpp-dev
>
> References
>
> Visible links
> 1. mailto:vpp-dev@lists.fd.io
> 2. https://lists.fd.io/mailman/listinfo/vpp-dev
> 3. mailto:vpp-dev@lists.fd.io
> 4. https://lists.fd.io/mailman/listinfo/vpp-dev
_______________________________________________
vpp-dev mailing list
vpp-dev@lists.fd.io
https://lists.fd.io/mailman/listinfo/vpp-dev
