Dear Paul, I think we should stop using python and move to rust :)
— Damjan > On 29.03.2021., at 21:20, Paul Vinciguerra <pvi...@vinciconsulting.com> wrote: > > Hi Damjan, > > Moving the files has tradeoffs. The relative value depends on the user's > workflow. Most tests have their own launcher at the bottom of the test_<foo> > file and at some point were able to run outside of the Makefile/run_tests. > Relocating the files essentially breaks that, but if it is not used by > anyone, that could be ok. We just need to clean up the code to remove those > remnants. > > If we were to clean up and package framework.py and related files as a python > module, then the tests could be run from anywhere in the file system. If > papi and the test framework were separated out of the repo, building tests > and applications would become 50x easier. Why? Because python would handle > the files and their locations/namespaces within the python venv. > > It would look something like: > $ cd <wherever_test_foo_lives> > $ python3 -m pip install . > installing <test_dependencies> > installing <framework> > installing vpp_papi > $ python3 ./test_<foo> > > None of these are insurmountable issues. We have to agree, and we have to > agree to not punt on the associated hygiene. > > I agree with Florin, I don't have a strong opinion as to where the files > reside. Unless, we can agree to move the tests to a separate repository, > then I'd prefer to have them all in a single directory. Why? Because it > would be trivial to run the stable release's tests against vpp master and > know what code I need to update across versions. > > While we are on the subject of tests, I would like to socialize a few ideas > for improvement. > Can we stop building packets in the test files? Yes, I know we need packets > to test against, but can we just have the test file pass an associated pcap > file, since that is what we send to the pg to run the test. A side effect > of this is that the need for scapy in the tests is reduced. There is a video > somewhere where all the different methods to sum up all the numbers from > 1..1_000_000 are compared in python and the final statement of the video was > to just use n(n + 1) / 2. > Can we make the testing packets more useful? We typically take a single > packet and replicate it a number of times to ensure that the packet takes > each path in the node. Additionally, the packet value choices aren't always > useful. When the source port == the dest port, for example, typos in the > code under test can be missed. Even better would be to increment ephemeral > port values. > I would like to see vpp add a stride width value (1,2,4,8) to a node's > registration info that was accessible to the test via an api, even if the api > were only in the unit test plugin. This way, the tests can calculate the > optimal packet stream characteristics, or at the very least warn that a given > test doesn't sufficiently test the component. > > Paul > > > On Fri, Mar 26, 2021 at 3:15 PM hemant via lists.fd.io <http://lists.fd.io/> > <hemant=mnkcg....@lists.fd.io <mailto:mnkcg....@lists.fd.io>> wrote: > Damjan, > > Thanks for clarifying - I agree with you on vnet tests. > > Hemant > > -----Original Message----- > From: vpp-dev@lists.fd.io <mailto:vpp-dev@lists.fd.io> <vpp-dev@lists.fd.io > <mailto:vpp-dev@lists.fd.io>> On Behalf Of Damjan Marion via lists.fd.io > <http://lists.fd.io/> > Sent: Friday, March 26, 2021 12:38 PM > To: Dave Wallace <dwallac...@gmail.com <mailto:dwallac...@gmail.com>> > Cc: vpp-dev <vpp-dev@lists.fd.io <mailto:vpp-dev@lists.fd.io>> > Subject: Re: [vpp-dev] keeping tests outside of src/ > > > > > On 25.03.2021., at 21:14, Dave Wallace <dwallac...@gmail.com > > <mailto:dwallac...@gmail.com>> wrote: > > > > Hi Damjan, > > > > This initiative originated with the wider adoption of plugin development at > > the request of Dave Barach to allow the development of plugins outside the > > VPP repo. After completing the job for plugins, there were several > > requests to extend that to all of the features. Presumably this was > > coupled with the desire to migrate feature source code from vnet into the > > plugin arena, but I don't recall all of the details of the discussion. > > OK, there are 2 different things. One is testing of out-of-tree plugins, > another one is VPP tree layout. > I fully agree that we need to support testing of out-of-tree plugins. Than > can be fixed as simple as ‘make test > TEST_DIR=/path/to/out-of-tree-plugin/test’. > > I don’t see how those two things relate. > > > > > Unfortunately, this effort stalled across several releases due to lack of > > cycles and I'm just now in the process of completing the job. > > > > I'm perfectly ok accepting a -2 for test code that maintainers prefer to > > leave in .../vpp/test, but I don't see the original requirement to > > co-locate plugin source & test code going away. So the majority of the > > feature source & test code will remain structured that way and the end > > result will be inconsistent at best. > > I’m not trying ot say we should -2, and I know that you submitted those > patches believing that this is the right thing to do. I am just under > impression that we are all not on the same page what is right thing to do. > > > > > > Personally, I think that it makes sense to continue to move features > > source, test code, and documentation to be co-located in a modular and > > consistent sub-tree structure. I also see value in migrating features out > > of vnet into the plugin sub-tree. > > I disagree here, I believe it should stay separate. But this is just my > opinion, I’m fine to be minority here, i just would like to know that we are > all on the same page and that whatever we decide we decided with good > understanding of implications. > > Implications may be: > - licensing implications like the current saga with scapy > - deciding if CMake should install test infra as part of vpp-dev packaging > - dealing with tests which cover multiple source code components or infra > > Mechanical move of file is the easiest part. As currently src/ is currently > one entity handled by CMake, throwing tests in without test infra being part > of src/ looks to me very broken. > > > > > > For what its worth, the changes to test/Makefile gather all of the source > > as soft links into the build tree (.../vpp/build-root/build-test/src), but > > I understand that is not the same your original plan. > > > > Thanks, > > -daw- > > > > On 3/25/2021 3:16 PM, Damjan Marion via lists.fd.io <http://lists.fd.io/> > > wrote: > >> Hi, > >> > >> It may be that it is not discussed or i was just ignorant, but I noticed > >> that there is ongoing activity to scatter tests all across the src/. > >> When I started "make test" long long time ago i intentionally put it > >> to separate tree following the pattern from other projects and to be > >> honest it makes me more sense that all tests are contained in the separate > >> tree. > >> > >> Are we sure that this test file scatter activity is right thing to do? > >> Anyone aware of any other project doing the same? > >> > >> — > >> Damjan > >> > >> > >> > >> > >> > >> > >> > > > > > > > > > > > >
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#19051): https://lists.fd.io/g/vpp-dev/message/19051 Mute This Topic: https://lists.fd.io/mt/81611239/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
