+1, simple is key here I think that we simply should remove that tap test as it falls under device driver tests which we don't test in make test framework.
PS I really don't like direction where test framework is going, originally it was very simple to write tests even without having python knowledge, it was simple copy/paste learn ty example exercise. Today you need to have advanced python skills even to understand what tests does. As i don't have any intention to grow my python skills, I can only declare myself as not capable to write any test case... -- Damjan > On 29 Dec 2019, at 20:09, Dave Barach via Lists.Fd.Io > <dbarach=cisco....@lists.fd.io> wrote: > > I would just as soon keep the story as simple as possible. “sudo make > test[-debug]” => run both test-runner and vpp privileged. “make test[-debug]” > means run neither one privileged. > > FWIW... Dave > > From: Paul Vinciguerra <pvi...@vinciconsulting.com > <mailto:pvi...@vinciconsulting.com>> > Sent: Saturday, December 28, 2019 12:22 PM > To: Dave Barach (dbarach) <dbar...@cisco.com <mailto:dbar...@cisco.com>> > Cc: Andrew Yourtchenko <ayour...@gmail.com <mailto:ayour...@gmail.com>>; > vpp-dev <vpp-dev@lists.fd.io <mailto:vpp-dev@lists.fd.io>> > Subject: Re: [vpp-dev] Discrepancies between CI jobs and s5ci > > Hi Dave. > > If you don't mind, would you clarify your statement further? Is it your > desire that 'make test[-debug]' be launched without privileges, or > is it that since VPP is a userspace application, you want to verify > functionality without any further dependencies on the kernel creeping in? > > The reason I'm asking is that because since you are aware of how the > framework tightly couples the permissions of the test runner and > the forked VPP process, you know that launching the test runner without > privileges is the current way we ensure that VPP runs without privileges. > We could have the runner run privileged and drop the capabilities of the > forked VPP process by default. > Today, 'make run' runs vpp under sudo. > > My thought is that a test needing additional capabilities could set a class > attribute of say capabilities = ['cap_net_admin'] and > we could add that capability for the single test suite and report > accordingly. Something like: > > ======================================================================================== > TAP Test Case > ======================================================================================== > [W173] Create TAP interface fails with insufficient permissions > 0.00 OK > [W173] Connect to existing TAP interface > 0.00 OK > ======================================================================================== > TAP Test Case > capabilities = ['cap_net_admin'] > ======================================================================================== > [W176] Create TAP interface > 0.22 OK > ======================================================================================== > > > If the runner is running unprivileged, we would skip: > > ======================================================================================== > TAP Test Case > capabilities = ['cap_net_admin'] > ======================================================================================== > [W176] Create TAP interface > 0.00 SKIP [ runner can't grant necessary capabilities. ] > ======================================================================================== > > On Sat, Dec 28, 2019 at 7:44 AM Dave Barach (dbarach) <dbar...@cisco.com > <mailto:dbar...@cisco.com>> wrote: > My $0.02: I would like “make test[-debug]” to work without privileges. > > Thanks... Dave > > From: vpp-dev@lists.fd.io <mailto:vpp-dev@lists.fd.io> <vpp-dev@lists.fd.io > <mailto:vpp-dev@lists.fd.io>> On Behalf Of Andrew Yourtchenko > Sent: Friday, December 27, 2019 7:21 PM > To: Paul Vinciguerra <pvi...@vinciconsulting.com > <mailto:pvi...@vinciconsulting.com>> > Cc: vpp-dev <vpp-dev@lists.fd.io <mailto:vpp-dev@lists.fd.io>> > Subject: Re: [vpp-dev] Discrepancies between CI jobs and s5ci > > > > > On 27 Dec 2019, at 22:48, Paul Vinciguerra <pvi...@vinciconsulting.com > <mailto:pvi...@vinciconsulting.com>> wrote: > > > Hi Andrew. > That was the test that caught my attention. I am of the opinion that s5ci > provides the behavior we expect. > > I personally has requested the tap test in its time to be moved to extended > tests, because of this. > > I presume the parties who made it appear in the standard tests will either > chime into this discussion with their reasoning or will make it disappear :-) > > > > I have a changeset that skips the test on s5ci instead of failing [0], but I > actually think that the proper test for the CI is a test where the test > fails, reporting insufficient permissions, since that is what an unprivileged > use would see. > > [0] https://gerrit.fd.io/r/c/vpp/+/24132 > <https://gerrit.fd.io/r/c/vpp/+/24132> > > One can not decide if the lack of privileges is due to misconfiguration or > due to intent, so there is no place for adaptivity and automagicness in this > context. > > If we all agree that “make test” is supposed to succeed in the unprivileged > environment then it should, end of story. With the exact same tests as in > privileged environment. > > And then for the more demanding tests there should be some sort of > “PRIVILEGED=yes make test” alternative that will invoke the tests that > require additional privileges and fail if it doesn’t get them. With > potentially more granular set of valued for “yes”. > > Testing is about determinism. > > —a > > > > On Fri, Dec 27, 2019 at 4:27 PM Andrew 👽 Yourtchenko <ayour...@gmail.com > <mailto:ayour...@gmail.com>> wrote: > I was under impression the “make test” was supposed to be passing when run > under the regular user. That means that the recently added tap tests will > make it fail: http://s5ci.myvpp.net/jobs/vpp-periodic-make-verify/ > <http://s5ci.myvpp.net/jobs/vpp-periodic-make-verify/> is only failures as of > 12:25 UTC 24th December. > > So, what is the direction the community wants to go ? > > --a > > > On 27 Dec 2019, at 17:22, Paul Vinciguerra <pvi...@vinciconsulting.com > <mailto:pvi...@vinciconsulting.com>> wrote: > > > I have been trying to figure out why I've been seeing different results > between the CI jobs and s5ci jobs. What I've discovered is that the CI jobs > are running as the root user in a fully privileged container, where s5ci runs > unprivileged. > I am aware that there has been a long-standing desire for tests to run as an > unprivileged user. Do we need to implement that constraint on the CI tests? > From my tests in the CI [0]: > Sanity test case passed. > Running as root(0):root(0) > Capabilities: ['cap_chown', 'cap_dac_override', 'cap_dac_read_search', > 'cap_fowner', 'cap_fsetid', 'cap_kill', 'cap_setgid', 'cap_setuid', > 'cap_setpcap', 'cap_linux_immutable', 'cap_net_bind_service', > 'cap_net_broadcast', 'cap_net_admin', 'cap_net_raw', 'cap_ipc_lock', > 'cap_ipc_owner', 'cap_sys_module', 'cap_sys_rawio', 'cap_sys_chroot', > 'cap_sys_ptrace', 'cap_sys_pacct', 'cap_sys_admin', 'cap_sys_boot', > 'cap_sys_nice', 'cap_sys_resource', 'cap_sys_time', 'cap_sys_tty_config', > 'cap_mknod', 'cap_lease', 'cap_audit_write', 'cap_audit_control', > 'cap_setfcap', 'cap_mac_override', 'cap_mac_admin', 'cap_syslog', > 'cap_wake_alarm', 'cap_block_suspend', 'cap_audit_read+eip'] > OS reports 72 available cpu(s). Free shm: 1,023.9921875MB > Found enough resources to run tests with 4 cores > [0] > https://logs.fd.io/production/vex-yul-rot-jenkins-1/vpp-verify-master-ubuntu1804/1331/console.log.gz > > <https://logs.fd.io/production/vex-yul-rot-jenkins-1/vpp-verify-master-ubuntu1804/1331/console.log.gz> > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > > View/Reply Online (#14981): https://lists.fd.io/g/vpp-dev/message/14981 > <https://lists.fd.io/g/vpp-dev/message/14981> > Mute This Topic: https://lists.fd.io/mt/69288694/675608 > <https://lists.fd.io/mt/69288694/675608> > Group Owner: vpp-dev+ow...@lists.fd.io <mailto:vpp-dev%2bow...@lists.fd.io> > Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub > <https://lists.fd.io/g/vpp-dev/unsub> [ayour...@gmail.com > <mailto:ayour...@gmail.com>] > -=-=-=-=-=-=-=-=-=-=-=- > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > > View/Reply Online (#14990): https://lists.fd.io/g/vpp-dev/message/14990 > <https://lists.fd.io/g/vpp-dev/message/14990> > Mute This Topic: https://lists.fd.io/mt/69288694/675642 > <https://lists.fd.io/mt/69288694/675642> > Group Owner: vpp-dev+ow...@lists.fd.io <mailto:vpp-dev+ow...@lists.fd.io> > Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub > <https://lists.fd.io/g/vpp-dev/unsub> [dmar...@me.com > <mailto:dmar...@me.com>] > -=-=-=-=-=-=-=-=-=-=-=-
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#14991): https://lists.fd.io/g/vpp-dev/message/14991 Mute This Topic: https://lists.fd.io/mt/69288694/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
