Re: [vpp-dev] vpp make test for verify - are we there yet ? [awoken]

Thu, 05 Jan 2017 05:59:49 -0800 

Yep, seems to be it.
Not sure how to handle this forward. Testing with python is not ideal
for timing checks...
A better test would be to isolate the BFD node from VPP, mock the VPP
infrastructure (node scheduling) and verify that it is sending the frames
out in a timely fashion (all this in C, emulating CPU clock).
But the effort to do that is magnitudes higher as compared to writing
the python test..

Maciek, Damjan, thoughts?

Thanks,
Klement

Quoting Andrew šŸ‘½ Yourtchenko (2017-01-05 14:48:03)
> For me on a not very fast xhyve VM with ubuntu 16.04 (my macbook pro)
> on master the BFD tests appear racy, see below.
> 
> Is that supposed to happen ?
> 
> My results of "make test" on master:
> 
> 
> ======================================================================
> ERROR: verify session goes down after inactivity
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File "/home/ubuntu/vpp/test/test_bfd.py", line 258, in test_conn_down
>     self.bfd_session_up()
>   File "/home/ubuntu/vpp/test/test_bfd.py", line 199, in bfd_session_up
>     p, ttp = self.wait_for_bfd_packet()
>   File "/home/ubuntu/vpp/test/test_bfd.py", line 184, in wait_for_bfd_packet
>     p = self.pg0.wait_for_packet(timeout=timeout)
>   File "/home/ubuntu/vpp/test/vpp_pg_interface.py", line 271, in 
> wait_for_packet
>     "timeout" % self.out_path)
> Exception: Capture file
> /tmp/vpp-unittest-BFDTestCase-9GSnEV/pg0_out.pcap did not appear
> within timeout
> 
> ======================================================================
> ERROR: hold BFD session up
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File "/home/ubuntu/vpp/test/test_bfd.py", line 251, in test_hold_up
>     self.bfd_session_up()
>   File "/home/ubuntu/vpp/test/test_bfd.py", line 199, in bfd_session_up
>     p, ttp = self.wait_for_bfd_packet()
>   File "/home/ubuntu/vpp/test/test_bfd.py", line 184, in wait_for_bfd_packet
>     p = self.pg0.wait_for_packet(timeout=timeout)
>   File "/home/ubuntu/vpp/test/vpp_pg_interface.py", line 271, in 
> wait_for_packet
>     "timeout" % self.out_path)
> Exception: Capture file
> /tmp/vpp-unittest-BFDTestCase-9GSnEV/pg0_out.pcap did not appear
> within timeout
> 
> ======================================================================
> ERROR: immediately honor remote min rx reduction
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File "/home/ubuntu/vpp/test/test_bfd.py", line 295, in
> test_immediate_remote_min_rx_reduce
>     self.wait_for_bfd_packet()
>   File "/home/ubuntu/vpp/test/test_bfd.py", line 184, in wait_for_bfd_packet
>     p = self.pg0.wait_for_packet(timeout=timeout)
>   File "/home/ubuntu/vpp/test/vpp_pg_interface.py", line 297, in 
> wait_for_packet
>     raise Exception("Packet didn't arrive within timeout")
> Exception: Packet didn't arrive within timeout
> 
> ======================================================================
> ERROR: bring BFD session up
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File "/home/ubuntu/vpp/test/test_bfd.py", line 213, in test_session_up
>     self.bfd_session_up()
>   File "/home/ubuntu/vpp/test/test_bfd.py", line 199, in bfd_session_up
>     p, ttp = self.wait_for_bfd_packet()
>   File "/home/ubuntu/vpp/test/test_bfd.py", line 184, in wait_for_bfd_packet
>     p = self.pg0.wait_for_packet(timeout=timeout)
>   File "/home/ubuntu/vpp/test/vpp_pg_interface.py", line 271, in 
> wait_for_packet
>     "timeout" % self.out_path)
> Exception: Capture file
> /tmp/vpp-unittest-BFDTestCase-9GSnEV/pg0_out.pcap did not appear
> within timeout
> 
> ======================================================================
> FAIL: verify slow periodic control frames while session down
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File "/home/ubuntu/vpp/test/test_bfd.py", line 228, in test_slow_timer
>     after - before, 0.70, 1.05, "time between slow packets")
>   File "/home/ubuntu/vpp/test/framework.py", line 542, in assert_in_range
>     self.assertTrue(expected_min <= real_value <= expected_max, msg)
> AssertionError: Invalid time between slow packets: 1.12259793282 out
> of range <0.7,1.05>
> 
> ======================================================================
> FAIL: Load Balancer IP4 GRE4
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File "/home/ubuntu/vpp/test/test_lb.py", line 159, in test_lb_ip4_gre4
>     self.checkCapture(gre4=True, isv4=True)
>   File "/home/ubuntu/vpp/test/test_lb.py", line 98, in checkCapture
>     self.pg0.assert_nothing_captured()
>   File "/home/ubuntu/vpp/test/vpp_pg_interface.py", line 228, in
> assert_nothing_captured
>     self.name)
> AssertionError: Non-empty capture file present for interface pg0
> 
> Ran 66 tests in 95.385s
> 
> FAILED (failures=2, errors=4, skipped=5)
> 
> Make retest:
> 
> 
> 
> ======================================================================
> ERROR: immediately honor remote min rx reduction
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File "/home/ubuntu/vpp/test/test_bfd.py", line 294, in
> test_immediate_remote_min_rx_reduce
>     self.bfd_session_up()
>   File "/home/ubuntu/vpp/test/test_bfd.py", line 199, in bfd_session_up
>     p, ttp = self.wait_for_bfd_packet()
>   File "/home/ubuntu/vpp/test/test_bfd.py", line 184, in wait_for_bfd_packet
>     p = self.pg0.wait_for_packet(timeout=timeout)
>   File "/home/ubuntu/vpp/test/vpp_pg_interface.py", line 271, in 
> wait_for_packet
>     "timeout" % self.out_path)
> Exception: Capture file
> /tmp/vpp-unittest-BFDTestCase-eJHgkH/pg0_out.pcap did not appear
> within timeout
> 
> ======================================================================
> ERROR: large remote RequiredMinRxInterval
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File "/home/ubuntu/vpp/test/test_bfd.py", line 270, in
> test_large_required_min_rx
>     self.bfd_session_up()
>   File "/home/ubuntu/vpp/test/test_bfd.py", line 199, in bfd_session_up
>     p, ttp = self.wait_for_bfd_packet()
>   File "/home/ubuntu/vpp/test/test_bfd.py", line 184, in wait_for_bfd_packet
>     p = self.pg0.wait_for_packet(timeout=timeout)
>   File "/home/ubuntu/vpp/test/vpp_pg_interface.py", line 271, in 
> wait_for_packet
>     "timeout" % self.out_path)
> Exception: Capture file
> /tmp/vpp-unittest-BFDTestCase-eJHgkH/pg0_out.pcap did not appear
> within timeout
> 
> ======================================================================
> ERROR: bring BFD session up
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File "/home/ubuntu/vpp/test/test_bfd.py", line 213, in test_session_up
>     self.bfd_session_up()
>   File "/home/ubuntu/vpp/test/test_bfd.py", line 199, in bfd_session_up
>     p, ttp = self.wait_for_bfd_packet()
>   File "/home/ubuntu/vpp/test/test_bfd.py", line 184, in wait_for_bfd_packet
>     p = self.pg0.wait_for_packet(timeout=timeout)
>   File "/home/ubuntu/vpp/test/vpp_pg_interface.py", line 271, in 
> wait_for_packet
>     "timeout" % self.out_path)
> Exception: Capture file
> /tmp/vpp-unittest-BFDTestCase-eJHgkH/pg0_out.pcap did not appear
> within timeout
> 
> Ran 66 tests in 96.297s
> 
> FAILED (errors=3, skipped=5)
> 
> 
> 
> ======================================================================
> ERROR: verify session goes down after inactivity
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File "/home/ubuntu/vpp/test/test_bfd.py", line 258, in test_conn_down
>     self.bfd_session_up()
>   File "/home/ubuntu/vpp/test/test_bfd.py", line 199, in bfd_session_up
>     p, ttp = self.wait_for_bfd_packet()
>   File "/home/ubuntu/vpp/test/test_bfd.py", line 184, in wait_for_bfd_packet
>     p = self.pg0.wait_for_packet(timeout=timeout)
>   File "/home/ubuntu/vpp/test/vpp_pg_interface.py", line 271, in 
> wait_for_packet
>     "timeout" % self.out_path)
> Exception: Capture file
> /tmp/vpp-unittest-BFDTestCase-ubN84d/pg0_out.pcap did not appear
> within timeout
> 
> ======================================================================
> ERROR: hold BFD session up
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File "/home/ubuntu/vpp/test/test_bfd.py", line 251, in test_hold_up
>     self.bfd_session_up()
>   File "/home/ubuntu/vpp/test/test_bfd.py", line 199, in bfd_session_up
>     p, ttp = self.wait_for_bfd_packet()
>   File "/home/ubuntu/vpp/test/test_bfd.py", line 184, in wait_for_bfd_packet
>     p = self.pg0.wait_for_packet(timeout=timeout)
>   File "/home/ubuntu/vpp/test/vpp_pg_interface.py", line 271, in 
> wait_for_packet
>     "timeout" % self.out_path)
> Exception: Capture file
> /tmp/vpp-unittest-BFDTestCase-ubN84d/pg0_out.pcap did not appear
> within timeout
> 
> ======================================================================
> FAIL: immediately honor remote min rx reduction
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File "/home/ubuntu/vpp/test/test_bfd.py", line 306, in
> test_immediate_remote_min_rx_reduce
>     "time between BFD packets")
>   File "/home/ubuntu/vpp/test/framework.py", line 542, in assert_in_range
>     self.assertTrue(expected_min <= real_value <= expected_max, msg)
> AssertionError: Invalid time between BFD packets: 0.111872911453 out
> of range <0.0675,0.11>
> 
> Ran 66 tests in 97.858s
> 
> FAILED (failures=1, errors=2, skipped=5)
> 
> 
> On 1/3/17, Maciek Konstantynowicz (mkonstan) <mkons...@cisco.com> wrote:
> > // Typed this email before the live discussion on vpp call just now - still
> > sending it out :)
> >
> > Hello, After the previous ver of this thread went into a frenzy of emails
> > and fixes in vpp make test code,
> > I wanted to re-check the situation.
> >
> > And as it is a New Year, Iā€™m not lazy anymore, and did run make test in both
> > vpp branches :)
> > vpp master branch works, stable/1701 does not..
> >
> > 1. master
> >     Ran 65 tests in 90.838s
> >
> >     OK (skipped=5)
> >     make[1]: Leaving directory `/home/maciek/src/vpp2/test'
> >
> > 2. stable/1701
> >   ======================================================================
> >   ERROR: GRE tunnel Tests
> >   Exception: Capture file did not appear within timeout
> >   ======================================================================
> >   ERROR: GRE tunnel L2 Tests
> >   Exception: Capture file did not appear within timeout
> >   ======================================================================
> >   ERROR: MPLS Local Label Binding test
> >   AttributeError: label
> >   ======================================================================
> >   ERROR: MPLS label imposition test
> >   IndexError: Layer [IP] not found
> >   ======================================================================
> >   ERROR: MPLS label swap tests
> >   AttributeError: label
> >   ======================================================================
> >   ERROR: MPLS Tunnel Tests
> >   ----------------------------------------------------------------------
> >   IndexError: Layer [IP] not found
> >   Ran 64 tests in 59.980s
> >
> >   FAILED (errors=6, skipped=5)
> >   make[1]: *** [test] Error 1
> >   make[1]: Leaving directory `/home/maciek/src/vpp2/test'
> >   make: *** [test] Error 2
> >
> > Fix?
> >
> > -Maciek
> >
> >
> >
> > On 12 Dec 2016, at 16:19, Maciek Konstantynowicz (mkonstan)
> > <mkons...@cisco.com<mailto:mkons...@cisco.com>> wrote:
> >
> > Hello, Does anyone know if vpp make test is back on track to be ready to be
> > used for vpp make verify jobs on a per patch basis?
> >
> > Being lazy I know - cause I could run it myself :)
> >
> > -Maciek
> >
> >
> >
> _______________________________________________
> vpp-dev mailing list
> vpp-dev@lists.fd.io
> https://lists.fd.io/mailman/listinfo/vpp-dev
_______________________________________________
vpp-dev mailing list
vpp-dev@lists.fd.io
https://lists.fd.io/mailman/listinfo/vpp-dev
          • ... Klement Sekera -X (ksekera - PANTHEON TECHNOLOGIES at Cisco)
            • ... Dave Wallace
            • ... Neale Ranns (nranns)
            • ... Dave Wallace
            • ... Klement Sekera -X (ksekera - PANTHEON TECHNOLOGIES at Cisco)
            • ... Neale Ranns (nranns)
            • ... Klement Sekera -X (ksekera - PANTHEON TECHNOLOGIES at Cisco)
            • ... Neale Ranns (nranns)
        • ... Klement Sekera -X (ksekera - PANTHEON TECHNOLOGIES at Cisco)
  • Re: [vpp-dev]... Andrew šŸ‘½ Yourtchenko
    • Re: [vpp... Klement Sekera -X (ksekera - PANTHEON TECHNOLOGIES at Cisco)

Reply via email to