Re: [vpp-dev] vpp make test for verify - are we there yet ? [awoken]

Andrew 👽 Yourtchenko Thu, 05 Jan 2017 05:49:23 -0800

For me on a not very fast xhyve VM with ubuntu 16.04 (my macbook pro)
on master the BFD tests appear racy, see below.


Is that supposed to happen ?

My results of "make test" on master:


======================================================================
ERROR: verify session goes down after inactivity
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ubuntu/vpp/test/test_bfd.py", line 258, in test_conn_down
    self.bfd_session_up()
  File "/home/ubuntu/vpp/test/test_bfd.py", line 199, in bfd_session_up
    p, ttp = self.wait_for_bfd_packet()
  File "/home/ubuntu/vpp/test/test_bfd.py", line 184, in wait_for_bfd_packet
    p = self.pg0.wait_for_packet(timeout=timeout)
  File "/home/ubuntu/vpp/test/vpp_pg_interface.py", line 271, in wait_for_packet
    "timeout" % self.out_path)
Exception: Capture file
/tmp/vpp-unittest-BFDTestCase-9GSnEV/pg0_out.pcap did not appear
within timeout

======================================================================
ERROR: hold BFD session up
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ubuntu/vpp/test/test_bfd.py", line 251, in test_hold_up
    self.bfd_session_up()
  File "/home/ubuntu/vpp/test/test_bfd.py", line 199, in bfd_session_up
    p, ttp = self.wait_for_bfd_packet()
  File "/home/ubuntu/vpp/test/test_bfd.py", line 184, in wait_for_bfd_packet
    p = self.pg0.wait_for_packet(timeout=timeout)
  File "/home/ubuntu/vpp/test/vpp_pg_interface.py", line 271, in wait_for_packet
    "timeout" % self.out_path)
Exception: Capture file
/tmp/vpp-unittest-BFDTestCase-9GSnEV/pg0_out.pcap did not appear
within timeout

======================================================================
ERROR: immediately honor remote min rx reduction
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ubuntu/vpp/test/test_bfd.py", line 295, in
test_immediate_remote_min_rx_reduce
    self.wait_for_bfd_packet()
  File "/home/ubuntu/vpp/test/test_bfd.py", line 184, in wait_for_bfd_packet
    p = self.pg0.wait_for_packet(timeout=timeout)
  File "/home/ubuntu/vpp/test/vpp_pg_interface.py", line 297, in wait_for_packet
    raise Exception("Packet didn't arrive within timeout")
Exception: Packet didn't arrive within timeout

======================================================================
ERROR: bring BFD session up
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ubuntu/vpp/test/test_bfd.py", line 213, in test_session_up
    self.bfd_session_up()
  File "/home/ubuntu/vpp/test/test_bfd.py", line 199, in bfd_session_up
    p, ttp = self.wait_for_bfd_packet()
  File "/home/ubuntu/vpp/test/test_bfd.py", line 184, in wait_for_bfd_packet
    p = self.pg0.wait_for_packet(timeout=timeout)
  File "/home/ubuntu/vpp/test/vpp_pg_interface.py", line 271, in wait_for_packet
    "timeout" % self.out_path)
Exception: Capture file
/tmp/vpp-unittest-BFDTestCase-9GSnEV/pg0_out.pcap did not appear
within timeout

======================================================================
FAIL: verify slow periodic control frames while session down
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ubuntu/vpp/test/test_bfd.py", line 228, in test_slow_timer
    after - before, 0.70, 1.05, "time between slow packets")
  File "/home/ubuntu/vpp/test/framework.py", line 542, in assert_in_range
    self.assertTrue(expected_min <= real_value <= expected_max, msg)
AssertionError: Invalid time between slow packets: 1.12259793282 out
of range <0.7,1.05>

======================================================================
FAIL: Load Balancer IP4 GRE4
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ubuntu/vpp/test/test_lb.py", line 159, in test_lb_ip4_gre4
    self.checkCapture(gre4=True, isv4=True)
  File "/home/ubuntu/vpp/test/test_lb.py", line 98, in checkCapture
    self.pg0.assert_nothing_captured()
  File "/home/ubuntu/vpp/test/vpp_pg_interface.py", line 228, in
assert_nothing_captured
    self.name)
AssertionError: Non-empty capture file present for interface pg0

Ran 66 tests in 95.385s

FAILED (failures=2, errors=4, skipped=5)

Make retest:



======================================================================
ERROR: immediately honor remote min rx reduction
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ubuntu/vpp/test/test_bfd.py", line 294, in
test_immediate_remote_min_rx_reduce
    self.bfd_session_up()
  File "/home/ubuntu/vpp/test/test_bfd.py", line 199, in bfd_session_up
    p, ttp = self.wait_for_bfd_packet()
  File "/home/ubuntu/vpp/test/test_bfd.py", line 184, in wait_for_bfd_packet
    p = self.pg0.wait_for_packet(timeout=timeout)
  File "/home/ubuntu/vpp/test/vpp_pg_interface.py", line 271, in wait_for_packet
    "timeout" % self.out_path)
Exception: Capture file
/tmp/vpp-unittest-BFDTestCase-eJHgkH/pg0_out.pcap did not appear
within timeout

======================================================================
ERROR: large remote RequiredMinRxInterval
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ubuntu/vpp/test/test_bfd.py", line 270, in
test_large_required_min_rx
    self.bfd_session_up()
  File "/home/ubuntu/vpp/test/test_bfd.py", line 199, in bfd_session_up
    p, ttp = self.wait_for_bfd_packet()
  File "/home/ubuntu/vpp/test/test_bfd.py", line 184, in wait_for_bfd_packet
    p = self.pg0.wait_for_packet(timeout=timeout)
  File "/home/ubuntu/vpp/test/vpp_pg_interface.py", line 271, in wait_for_packet
    "timeout" % self.out_path)
Exception: Capture file
/tmp/vpp-unittest-BFDTestCase-eJHgkH/pg0_out.pcap did not appear
within timeout

======================================================================
ERROR: bring BFD session up
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ubuntu/vpp/test/test_bfd.py", line 213, in test_session_up
    self.bfd_session_up()
  File "/home/ubuntu/vpp/test/test_bfd.py", line 199, in bfd_session_up
    p, ttp = self.wait_for_bfd_packet()
  File "/home/ubuntu/vpp/test/test_bfd.py", line 184, in wait_for_bfd_packet
    p = self.pg0.wait_for_packet(timeout=timeout)
  File "/home/ubuntu/vpp/test/vpp_pg_interface.py", line 271, in wait_for_packet
    "timeout" % self.out_path)
Exception: Capture file
/tmp/vpp-unittest-BFDTestCase-eJHgkH/pg0_out.pcap did not appear
within timeout

Ran 66 tests in 96.297s

FAILED (errors=3, skipped=5)



======================================================================
ERROR: verify session goes down after inactivity
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ubuntu/vpp/test/test_bfd.py", line 258, in test_conn_down
    self.bfd_session_up()
  File "/home/ubuntu/vpp/test/test_bfd.py", line 199, in bfd_session_up
    p, ttp = self.wait_for_bfd_packet()
  File "/home/ubuntu/vpp/test/test_bfd.py", line 184, in wait_for_bfd_packet
    p = self.pg0.wait_for_packet(timeout=timeout)
  File "/home/ubuntu/vpp/test/vpp_pg_interface.py", line 271, in wait_for_packet
    "timeout" % self.out_path)
Exception: Capture file
/tmp/vpp-unittest-BFDTestCase-ubN84d/pg0_out.pcap did not appear
within timeout

======================================================================
ERROR: hold BFD session up
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ubuntu/vpp/test/test_bfd.py", line 251, in test_hold_up
    self.bfd_session_up()
  File "/home/ubuntu/vpp/test/test_bfd.py", line 199, in bfd_session_up
    p, ttp = self.wait_for_bfd_packet()
  File "/home/ubuntu/vpp/test/test_bfd.py", line 184, in wait_for_bfd_packet
    p = self.pg0.wait_for_packet(timeout=timeout)
  File "/home/ubuntu/vpp/test/vpp_pg_interface.py", line 271, in wait_for_packet
    "timeout" % self.out_path)
Exception: Capture file
/tmp/vpp-unittest-BFDTestCase-ubN84d/pg0_out.pcap did not appear
within timeout

======================================================================
FAIL: immediately honor remote min rx reduction
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ubuntu/vpp/test/test_bfd.py", line 306, in
test_immediate_remote_min_rx_reduce
    "time between BFD packets")
  File "/home/ubuntu/vpp/test/framework.py", line 542, in assert_in_range
    self.assertTrue(expected_min <= real_value <= expected_max, msg)
AssertionError: Invalid time between BFD packets: 0.111872911453 out
of range <0.0675,0.11>

Ran 66 tests in 97.858s

FAILED (failures=1, errors=2, skipped=5)


On 1/3/17, Maciek Konstantynowicz (mkonstan) <mkons...@cisco.com> wrote:
> // Typed this email before the live discussion on vpp call just now - still
> sending it out :)
>
> Hello, After the previous ver of this thread went into a frenzy of emails
> and fixes in vpp make test code,
> I wanted to re-check the situation.
>
> And as it is a New Year, I’m not lazy anymore, and did run make test in both
> vpp branches :)
> vpp master branch works, stable/1701 does not..
>
> 1. master
>     Ran 65 tests in 90.838s
>
>     OK (skipped=5)
>     make[1]: Leaving directory `/home/maciek/src/vpp2/test'
>
> 2. stable/1701
>   ======================================================================
>   ERROR: GRE tunnel Tests
>   Exception: Capture file did not appear within timeout
>   ======================================================================
>   ERROR: GRE tunnel L2 Tests
>   Exception: Capture file did not appear within timeout
>   ======================================================================
>   ERROR: MPLS Local Label Binding test
>   AttributeError: label
>   ======================================================================
>   ERROR: MPLS label imposition test
>   IndexError: Layer [IP] not found
>   ======================================================================
>   ERROR: MPLS label swap tests
>   AttributeError: label
>   ======================================================================
>   ERROR: MPLS Tunnel Tests
>   ----------------------------------------------------------------------
>   IndexError: Layer [IP] not found
>   Ran 64 tests in 59.980s
>
>   FAILED (errors=6, skipped=5)
>   make[1]: *** [test] Error 1
>   make[1]: Leaving directory `/home/maciek/src/vpp2/test'
>   make: *** [test] Error 2
>
> Fix?
>
> -Maciek
>
>
>
> On 12 Dec 2016, at 16:19, Maciek Konstantynowicz (mkonstan)
> <mkons...@cisco.com<mailto:mkons...@cisco.com>> wrote:
>
> Hello, Does anyone know if vpp make test is back on track to be ready to be
> used for vpp make verify jobs on a per patch basis?
>
> Being lazy I know - cause I could run it myself :)
>
> -Maciek
>
>
>
_______________________________________________
vpp-dev mailing list
vpp-dev@lists.fd.io
https://lists.fd.io/mailman/listinfo/vpp-dev

Re: [vpp-dev] vpp make test for verify - are we there yet ? [awoken]

Reply via email to