Hi Dave, my research says that the timestamp is inside the .pyc file and that the modification time of the file is ignored. I tried to fool python by having a source compiled, the changing the source file and touching the .pyc file to see if it'll use stale bytecode, but it didn't.
Either way - if this works, then that's great. Just one comment - could you please add a comment describing why the sleep is needed? Thanks, Klement Quoting Dave Wallace (2017-01-27 03:16:25) > Klement/Neale, > > I finally got back to debugging this issue. After doing a make > test-wipe, then make test. The patched python files and their > corresponding .pyc files had the same timestamp with a 1-second > resolution. I deleted all .pyc files and then all of the tests passed. > After doing some research, I confirmed that python uses a 1-second > resolution when checking whether to recompile the *.pyc file. > > I have verified that the following patch resolves this issue both on > Ubuntu 16.04 in Vagrant and Ubuntu 16.10 on bare metal: > > https://gerrit.fd.io/r/#/c/4896/ > > Thanks, > -daw- > > On 01/05/2017 03:54 AM, Klement Sekera -X (ksekera - PANTHEON > TECHNOLOGIES at Cisco) wrote: > > Dave, there is no sign of test framework installing/patching scapy in > > the log you provided. > > > > Could you please do in the same workspace > > > > make test-wipe > > > > and then run the make test again? I don't think it'll actually help us, > > because there should be no way for a python inside virtualenv to get its > > hands on outside libs (unless there is a bug of course), but let's see > > what happens... > > > > Thanks, > > Klement > > > > Quoting Dave Wallace (2017-01-04 19:19:04) > >> Neale, > >> > >> I could not find scapy installed in the normal path (i.e. "which > >> scapy"), > >> but I'm not sure if it is directly executable. You can find the "V=1 > >> TEST=mpls make test" output on > >> vagrant+virtualbox:puppetlabs/ubuntu-16.04.64-nocm in this pastebin > >> doc: > >> [1]http://pastebin.com/uGAUREhK > >> > >> Thanks, > >> -daw- > >> > >> On 1/4/2017 10:59 AM, Neale Ranns (nranns) wrote: > >> > >> > >> > >> Hi, > >> > >> > >> > >> The reason the MPLS tests are failing is because scapy cannot decode > >> an > >> MPLS label stack (output from some .show() instrumentation); > >> > >> > >> > >> <class 'scapy.layers.l2.Ether'> > >> > >> ###[ Ethernet ]### > >> > >> dst = 02:01:00:00:ff:02 > >> > >> src = 02:fe:e5:05:0e:13 > >> > >> type = 0x8847 > >> > >> ###[ MPLS ]### > >> > >> label = 33L > >> > >> cos = 0L > >> > >> s = 0L <<<<<< NON End-of-stack > >> > >> ttl = 254 > >> > >> ###[ Padding ]### > >> > >> load = '\x00\x061\xffE\x00\x00#\x00\x01\x00\x00@\x11 > >> > >> \xa6\xac\x10\x01\x01\xac\x10\x01\x02\x04\xd2\x04\xd2\x00\x0f\xd0\x92257 > >> 1 1' > >> > >> > >> > >> we patch scapy for this purpose, see; > >> > >> <ROOT>/test/patches/scapy-2.3.3/mpls.py.patch > >> > >> > >> > >> on my vagrant 14.04 there is no other scapy installation, this patch > >> applies fine and all MPLS tests pass. > >> > >> On a UCS 14.04, with another scapy installation: > >> > >> /usr/local/lib/python2.7/dist-packages/scapy/contrib/mpls.py > >> > >> the MPLS test fail. > >> > >> On a UCS 16.04 no scpay installation, tests pass. > >> > >> > >> > >> Do you have scapy installed in the machines on which the tests > >> fail/pass? Or am I barking up the wrong virtualenv tree J > >> > >> > >> > >> Thanks, > >> > >> neale > >> > >> > >> > >> > >> > >> > >> > >> From: Dave Wallace [2]<dwallac...@gmail.com> > >> Date: Wednesday, 4 January 2017 at 15:38 > >> To: "Klement Sekera -X (ksekera - PANTHEON TECHNOLOGIES at Cisco)" > >> [3]<ksek...@cisco.com>, "Maciek Konstantynowicz (mkonstan)" > >> [4]<mkons...@cisco.com>, "Neale Ranns (nranns)" > >> [5]<nra...@cisco.com> > >> Cc: [6]"csit-...@lists.fd.io" [7]<csit-...@lists.fd.io>, > >> [8]"vpp-dev@lists.fd.io" [9]<vpp-dev@lists.fd.io> > >> Subject: Re: [csit-dev] [vpp-dev] vpp make test for verify - are we > >> there yet ? [awoken] > >> > >> > >> > >> Klement, > >> > >> I'm in the process of modifying the base Vagrantfile/build scripts > >> to > >> replace building the VPP native package and installing it in the > >> VM to > >> simply running "make test" which takes much less time. All of the > >> test runs that I have performed are clean builds on a virgin git > >> clone > >> repo. > >> > >> That being said, I'll give the "git clean -dfX */" a try and see > >> if I > >> get different results on the 2nd pass and report back. > >> > >> Thanks, > >> -daw- > >> > >> On 1/4/17 10:23 AM, Klement Sekera -X (ksekera - PANTHEON > >> TECHNOLOGIES > >> at Cisco) wrote: > >> > >> Hi Dave, > >> > >> > >> > >> could you please try running git clean -dfX */ in a workspace, where you > >> see > >> > >> the failures, and then givet it another try? For me this seems to clear > >> issues > >> > >> up. Damjans recent changes might have caused (quoting Damjan) "major > >> > >> disturbances in the force", leaving build artifacts behind. > >> > >> > >> > >> Just be sure to stash/commit any changes you have in the workspace since > >> > >> the git clean nukes everything unknown from the ws. > >> > >> > >> > >> Thanks, > >> > >> Klement > >> > >> > >> > >> Quoting Dave Wallace (2017-01-04 16:12:38) > >> > >> As I mentioned on the VPP call yesterday, I'm seeing what appears to > >> be > >> > >> the same GRE test failures on Ubuntu 16.10 (bare metal). I'm also > >> seeing > >> > >> the same failures on vagrant+virtualbox with > >> > >> puppetlabs/ubuntu-16.04.64-nocm. I also am getting failures on 3 out > >> of 4 > >> > >> MPLS tests on both environments. > >> > >> > >> > >> However, all tests pass on vagrant+virtualbox with > >> > >> puppetlabs/ubuntu-14.04.64-nocm. > >> > >> > >> > >> Thanks, > >> > >> -daw- > >> > >> > >> > >> On 1/4/17 9:47 AM, Maciek Konstantynowicz (mkonstan) wrote: > >> > >> > >> > >> baremetal trusty: > >> > >> > >> > >> $ lsb_release -a > >> > >> No LSB modules are available. > >> > >> Distributor ID: Ubuntu > >> > >> Description: Ubuntu 14.04.4 LTS > >> > >> Release: 14.04 > >> > >> Codename: trusty > >> > >> $ uname -a > >> > >> Linux homes1 3.16.0-77-generic #99~14.04.1-Ubuntu SMP Tue Jun 28 > >> 19:17:10 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux > >> > >> > >> > >> -Maciek > >> > >> > >> > >> > >> > >> On 4 Jan 2017, at 13:42, Neale Ranns (nranns) [1][10]<nra...@cisco.com> > >> wrote: > >> > >> > >> > >> > >> > >> Hi Klement, Maciek, > >> > >> > >> > >> What environment are you running the tests in? > >> > >> I get different results. Running in the default vagrant env provided in > >> 17.0. Everything passes apart from: > >> > >> > >> > >> ====================================================================== > >> > >> Bidirectional Forwarding Detection (BFD) > >> > >> ====================================================================== > >> > >> verify session goes down after inactivity OK > >> > >> hold BFD session up OK > >> > >> large remote RequiredMinRxInterval ERROR [ > >> temp dir used by test case: /tmp/vpp-unittest-BFDTestCase-F7O6hZ ] > >> > >> bring BFD session up ERROR [ > >> temp dir used by test case: /tmp/vpp-unittest-BFDTestCase-F7O6hZ ] > >> > >> verify slow periodic control frames while session down ERROR [ > >> temp dir used by test case: /tmp/vpp-unittest-BFDTestCase-F7O6hZ ] > >> > >> no packets when zero BFD RemoteMinRxInterval ERROR [ > >> temp dir used by test case: /tmp/vpp-unittest-BFDTestCase-F7O6hZ ] > >> > >> > >> > >> details below. Different runs give different combinations of failures > >> from this BFD list. > >> > >> > >> > >> /neale > >> > >> > >> > >> ===================================================================== > >> > >> ERROR: large remote RequiredMinRxInterval > >> > >> ---------------------------------------------------------------------- > >> > >> Traceback (most recent call last): > >> > >> File "/vpp/test/test_bfd.py", line 257, in test_large_required_min_rx > >> > >> self.bfd_session_up() > >> > >> File "/vpp/test/test_bfd.py", line 219, in bfd_session_up > >> > >> p = self.wait_for_bfd_packet() > >> > >> File "/vpp/test/test_bfd.py", line 171, in wait_for_bfd_packet > >> > >> p = self.pg0.wait_for_packet(timeout=timeout) > >> > >> File "/vpp/test/vpp_pg_interface.py", line 217, in wait_for_packet > >> > >> self.wait_for_capture_file(timeout) > >> > >> File "/vpp/test/vpp_pg_interface.py", line 204, in > >> wait_for_capture_file > >> > >> raise Exception("Capture file did not appear within timeout") > >> > >> Exception: Capture file did not appear within timeout > >> > >> > >> > >> ====================================================================== > >> > >> ERROR: bring BFD session up > >> > >> ---------------------------------------------------------------------- > >> > >> Traceback (most recent call last): > >> > >> File "/vpp/test/test_bfd.py", line 234, in test_session_up > >> > >> self.bfd_session_up() > >> > >> File "/vpp/test/test_bfd.py", line 219, in bfd_session_up > >> > >> p = self.wait_for_bfd_packet() > >> > >> File "/vpp/test/test_bfd.py", line 171, in wait_for_bfd_packet > >> > >> p = self.pg0.wait_for_packet(timeout=timeout) > >> > >> File "/vpp/test/vpp_pg_interface.py", line 217, in wait_for_packet > >> > >> self.wait_for_capture_file(timeout) > >> > >> File "/vpp/test/vpp_pg_interface.py", line 204, in > >> wait_for_capture_file > >> > >> raise Exception("Capture file did not appear within timeout") > >> > >> Exception: Capture file did not appear within timeout > >> > >> > >> > >> ====================================================================== > >> > >> ERROR: verify slow periodic control frames while session down > >> > >> ---------------------------------------------------------------------- > >> > >> Traceback (most recent call last): > >> > >> File "/vpp/test/test_bfd.py", line 187, in test_slow_timer > >> > >> self.wait_for_bfd_packet() > >> > >> File "/vpp/test/test_bfd.py", line 171, in wait_for_bfd_packet > >> > >> p = self.pg0.wait_for_packet(timeout=timeout) > >> > >> File "/vpp/test/vpp_pg_interface.py", line 217, in wait_for_packet > >> > >> self.wait_for_capture_file(timeout) > >> > >> File "/vpp/test/vpp_pg_interface.py", line 204, in > >> wait_for_capture_file > >> > >> raise Exception("Capture file did not appear within timeout") > >> > >> Exception: Capture file did not appear within timeout > >> > >> > >> > >> ====================================================================== > >> > >> ERROR: no packets when zero BFD RemoteMinRxInterval > >> > >> ---------------------------------------------------------------------- > >> > >> Traceback (most recent call last): > >> > >> File "/vpp/test/test_bfd.py", line 201, in test_zero_remote_min_rx > >> > >> p = self.wait_for_bfd_packet() > >> > >> File "/vpp/test/test_bfd.py", line 171, in wait_for_bfd_packet > >> > >> p = self.pg0.wait_for_packet(timeout=timeout) > >> > >> File "/vpp/test/vpp_pg_interface.py", line 217, in wait_for_packet > >> > >> self.wait_for_capture_file(timeout) > >> > >> File "/vpp/test/vpp_pg_interface.py", line 204, in > >> wait_for_capture_file > >> > >> raise Exception("Capture file did not appear within timeout") > >> > >> Exception: Capture file did not appear within timeout > >> > >> > >> > >> Ran 64 tests in 84.161s > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> On 04/01/2017, 09:45, [2][11]"csit-dev-boun...@lists.fd.io on behalf of > >> Klement Sekera -X (ksekera - PANTHEON TECHNOLOGIES at Cisco)" > >> [3][12]<csit-dev-boun...@lists.fd.io on behalf of ksek...@cisco.com> wrote: > >> > >> > >> > >> I tried doing a git checkout origin/stable/1701 - is that the correct > >> > >> branch? On that one, the GRE indeed doesn't work, but I don't think > >> it's > >> > >> 'make test' issue, here's a sample from packet trace: > >> > >> > >> > >> ------------------- Start of thread 0 vpp_main ------------------- > >> > >> Packet 1 > >> > >> > >> > >> 00:00:07:119382: pg-input > >> > >> stream pcap1, 88 bytes > >> > >> current data 0, length 88, free-list 6, trace 0x0 > >> > >> IP4: 02:01:00:00:ff:02 -> 02:fe:39:17:74:65 > >> > >> GRE: 2.2.2.2 -> 172.16.1.1 > >> > >> tos 0x00, ttl 64, length 74, checksum 0xc96f > >> > >> fragment id 0x0001 > >> > >> GRE 0x0001 > >> > >> 00:00:07:119802: ethernet-input > >> > >> IP4: 02:01:00:00:ff:02 -> 02:fe:39:17:74:65 > >> > >> 00:00:07:120048: ip4-input > >> > >> GRE: 2.2.2.2 -> 172.16.1.1 > >> > >> tos 0x00, ttl 64, length 74, checksum 0xc96f > >> > >> fragment id 0x0001 > >> > >> GRE 0x0001 > >> > >> 00:00:07:120113: ip4-lookup > >> > >> fib 0 dpo-idx 5 flow hash: 0x00000000 > >> > >> GRE: 2.2.2.2 -> 172.16.1.1 > >> > >> tos 0x00, ttl 64, length 74, checksum 0xc96f > >> > >> fragment id 0x0001 > >> > >> GRE 0x0001 > >> > >> 00:00:07:120404: ip4-local > >> > >> GRE: 2.2.2.2 -> 172.16.1.1 > >> > >> tos 0x00, ttl 64, length 74, checksum 0xc96f > >> > >> fragment id 0x0001 > >> > >> GRE 0x0001 > >> > >> 00:00:07:120495: gre-input > >> > >> GRE: tunnel 0 len 74 src 2.2.2.2 dst 172.16.1.1 > >> > >> 00:00:07:120543: error-drop > >> > >> gre-input: unknown protocol > >> > >> > >> > >> all 50 packets printed in the packet trace share the same fate... > >> > >> > >> > >> Thanks, > >> > >> Klement > >> > >> > >> > >> Quoting Maciek Konstantynowicz (mkonstan) (2017-01-03 18:46:19) > >> > >> > >> > >> // Typed this email before the live discussion on vpp call just now - > >> > >> still sending it out :) > >> > >> Hello, After the previous ver of this thread went into a frenzy of > >> emails > >> > >> and fixes in vpp make test code, > >> > >> I wanted to re-check the situation. > >> > >> And as it is a New Year, I’m not lazy anymore, and did run make test > >> in > >> > >> both vpp branches :) > >> > >> vpp master branch works, stable/1701 does not.. > >> > >> 1. master > >> > >> Ran 65 tests in 90.838s > >> > >> OK (skipped=5) > >> > >> make[1]: Leaving directory `/home/maciek/src/vpp2/test' > >> > >> 2. stable/1701 > >> > >> > >> ====================================================================== > >> > >> ERROR: GRE tunnel Tests > >> > >> Exception: Capture file did not appear within timeout > >> > >> > >> ====================================================================== > >> > >> ERROR: GRE tunnel L2 Tests > >> > >> Exception: Capture file did not appear within timeout > >> > >> > >> ====================================================================== > >> > >> ERROR: MPLS Local Label Binding test > >> > >> AttributeError: label > >> > >> > >> ====================================================================== > >> > >> ERROR: MPLS label imposition test > >> > >> IndexError: Layer [IP] not found > >> > >> > >> ====================================================================== > >> > >> ERROR: MPLS label swap tests > >> > >> AttributeError: label > >> > >> > >> ====================================================================== > >> > >> ERROR: MPLS Tunnel Tests > >> > >> > >> ---------------------------------------------------------------------- > >> > >> IndexError: Layer [IP] not found > >> > >> Ran 64 tests in 59.980s > >> > >> FAILED (errors=6, skipped=5) > >> > >> make[1]: *** [test] Error 1 > >> > >> make[1]: Leaving directory `/home/maciek/src/vpp2/test' > >> > >> make: *** [test] Error 2 > >> > >> Fix? > >> > >> -Maciek > >> > >> > >> > >> On 12 Dec 2016, at 16:19, Maciek Konstantynowicz (mkonstan) > >> > >> [4][13]<[1]mkons...@cisco.com> wrote: > >> > >> Hello, Does anyone know if vpp make test is back on track to be > >> ready to > >> > >> be used for vpp make verify jobs on a per patch basis? > >> > >> Being lazy I know - cause I could run it myself :) > >> > >> -Maciek > >> > >> > >> > >> References > >> > >> > >> > >> Visible links > >> > >> 1. [5][14]mailto:mkons...@cisco.com > >> > >> > >> > >> _______________________________________________ > >> > >> csit-dev mailing list > >> > >> [[15]6]csit-...@lists.fd.io > >> > >> [7][16]https://lists.fd.io/mailman/listinfo/csit-dev > >> > >> > >> > >> > >> > >> _______________________________________________ > >> > >> csit-dev mailing list > >> > >> [[17]8]csit-...@lists.fd.io > >> > >> [9][18]https://lists.fd.io/mailman/listinfo/csit-dev > >> > >> > >> > >> References > >> > >> > >> > >> Visible links > >> > >> 1. [19]mailto:nra...@cisco.com > >> > >> 2. > >> [20]mailto:csit-dev-boun...@lists.fd.ioonbehalfofklementsekera-x(ksekera-pantheontechnologiesatcisco) > >> > >> 3. [21]mailto:csit-dev-boun...@lists.fd.ioonbehalfofksek...@cisco.com > >> > >> 4. mailto:[[22]1]mkons...@cisco.com > >> > >> 5. [23]mailto:mkons...@cisco.com > >> > >> 6. [24]mailto:csit-...@lists.fd.io > >> > >> 7. [25]https://lists.fd.io/mailman/listinfo/csit-dev > >> > >> 8. [26]mailto:csit-...@lists.fd.io > >> > >> 9. [27]https://lists.fd.io/mailman/listinfo/csit-dev > >> > >> > >> > >> References > >> > >> Visible links > >> 1. http://pastebin.com/uGAUREhK > >> 2. mailto:dwallac...@gmail.com > >> 3. mailto:ksek...@cisco.com > >> 4. mailto:mkons...@cisco.com > >> 5. mailto:nra...@cisco.com > >> 6. mailto:csit-...@lists.fd.io > >> 7. mailto:csit-...@lists.fd.io > >> 8. mailto:vpp-dev@lists.fd.io > >> 9. mailto:vpp-dev@lists.fd.io > >> 10. mailto:nra...@cisco.com > >> 11. > >> mailto:csit-dev-boun...@lists.fd.ioonbehalfofklementsekera-x%28ksekera-pantheontechnologiesatcisco%29 > >> 12. mailto:csit-dev-boun...@lists.fd.ioonbehalfofksek...@cisco.com > >> 13. mailto:[1]mkons...@cisco.com > >> 14. mailto:mkons...@cisco.com > >> 15. mailto:6]csit-...@lists.fd.io > >> 16. https://lists.fd.io/mailman/listinfo/csit-dev > >> 17. mailto:8]csit-...@lists.fd.io > >> 18. https://lists.fd.io/mailman/listinfo/csit-dev > >> 19. mailto:nra...@cisco.com > >> 20. > >> mailto:csit-dev-boun...@lists.fd.ioonbehalfofklementsekera-x%28ksekera-pantheontechnologiesatcisco%29 > >> 21. mailto:csit-dev-boun...@lists.fd.ioonbehalfofksek...@cisco.com > >> 22. mailto:1]mkons...@cisco.com > >> 23. mailto:mkons...@cisco.com > >> 24. mailto:csit-...@lists.fd.io > >> 25. https://lists.fd.io/mailman/listinfo/csit-dev > >> 26. mailto:csit-...@lists.fd.io > >> 27. https://lists.fd.io/mailman/listinfo/csit-dev > > _______________________________________________ vpp-dev mailing list vpp-dev@lists.fd.io https://lists.fd.io/mailman/listinfo/vpp-dev
