I agree with Damjan on this. Seems convoluted to comprehend and enhance the suite. I think, it isn’t very intuitive. But I guess we are tied to python to interface with the tools (scapy, trex, blah for traffic profiles)?
-- Regards, Balaji. From: <vpp-dev@lists.fd.io> on behalf of "Damjan Marion via Lists.Fd.Io" <dmarion=me....@lists.fd.io> Reply-To: "dmar...@me.com" <dmar...@me.com> Date: Sunday, December 29, 2019 at 11:21 AM To: "Dave Barach (dbarach)" <dbar...@cisco.com> Cc: "vpp-dev@lists.fd.io" <vpp-dev@lists.fd.io> Subject: Re: [vpp-dev] Discrepancies between CI jobs and s5ci +1, simple is key here I think that we simply should remove that tap test as it falls under device driver tests which we don't test in make test framework. PS I really don't like direction where test framework is going, originally it was very simple to write tests even without having python knowledge, it was simple copy/paste learn ty example exercise. Today you need to have advanced python skills even to understand what tests does. As i don't have any intention to grow my python skills, I can only declare myself as not capable to write any test case... -- Damjan On 29 Dec 2019, at 20:09, Dave Barach via Lists.Fd.Io <dbarach=cisco....@lists.fd.io<mailto:dbarach=cisco....@lists.fd.io>> wrote: I would just as soon keep the story as simple as possible. “sudo make test[-debug]” => run both test-runner and vpp privileged. “make test[-debug]” means run neither one privileged. FWIW... Dave From: Paul Vinciguerra <pvi...@vinciconsulting.com<mailto:pvi...@vinciconsulting.com>> Sent: Saturday, December 28, 2019 12:22 PM To: Dave Barach (dbarach) <dbar...@cisco.com<mailto:dbar...@cisco.com>> Cc: Andrew Yourtchenko <ayour...@gmail.com<mailto:ayour...@gmail.com>>; vpp-dev <vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io>> Subject: Re: [vpp-dev] Discrepancies between CI jobs and s5ci Hi Dave. If you don't mind, would you clarify your statement further? Is it your desire that 'make test[-debug]' be launched without privileges, or is it that since VPP is a userspace application, you want to verify functionality without any further dependencies on the kernel creeping in? The reason I'm asking is that because since you are aware of how the framework tightly couples the permissions of the test runner and the forked VPP process, you know that launching the test runner without privileges is the current way we ensure that VPP runs without privileges. We could have the runner run privileged and drop the capabilities of the forked VPP process by default. Today, 'make run' runs vpp under sudo. My thought is that a test needing additional capabilities could set a class attribute of say capabilities = ['cap_net_admin'] and we could add that capability for the single test suite and report accordingly. Something like: ======================================================================================== TAP Test Case ======================================================================================== [W173] Create TAP interface fails with insufficient permissions 0.00 OK [W173] Connect to existing TAP interface 0.00 OK ======================================================================================== TAP Test Case capabilities = ['cap_net_admin'] ======================================================================================== [W176] Create TAP interface 0.22 OK ======================================================================================== If the runner is running unprivileged, we would skip: ======================================================================================== TAP Test Case capabilities = ['cap_net_admin'] ======================================================================================== [W176] Create TAP interface 0.00 SKIP [ runner can't grant necessary capabilities. ] ======================================================================================== On Sat, Dec 28, 2019 at 7:44 AM Dave Barach (dbarach) <dbar...@cisco.com<mailto:dbar...@cisco.com>> wrote: My $0.02: I would like “make test[-debug]” to work without privileges. Thanks... Dave From: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> <vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io>> On Behalf Of Andrew Yourtchenko Sent: Friday, December 27, 2019 7:21 PM To: Paul Vinciguerra <pvi...@vinciconsulting.com<mailto:pvi...@vinciconsulting.com>> Cc: vpp-dev <vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io>> Subject: Re: [vpp-dev] Discrepancies between CI jobs and s5ci On 27 Dec 2019, at 22:48, Paul Vinciguerra <pvi...@vinciconsulting.com<mailto:pvi...@vinciconsulting.com>> wrote: Hi Andrew. That was the test that caught my attention. I am of the opinion that s5ci provides the behavior we expect. I personally has requested the tap test in its time to be moved to extended tests, because of this. I presume the parties who made it appear in the standard tests will either chime into this discussion with their reasoning or will make it disappear :-) I have a changeset that skips the test on s5ci instead of failing [0], but I actually think that the proper test for the CI is a test where the test fails, reporting insufficient permissions, since that is what an unprivileged use would see. [0] https://gerrit.fd.io/r/c/vpp/+/24132 One can not decide if the lack of privileges is due to misconfiguration or due to intent, so there is no place for adaptivity and automagicness in this context. If we all agree that “make test” is supposed to succeed in the unprivileged environment then it should, end of story. With the exact same tests as in privileged environment. And then for the more demanding tests there should be some sort of “PRIVILEGED=yes make test” alternative that will invoke the tests that require additional privileges and fail if it doesn’t get them. With potentially more granular set of valued for “yes”. Testing is about determinism. —a On Fri, Dec 27, 2019 at 4:27 PM Andrew 👽 Yourtchenko <ayour...@gmail.com<mailto:ayour...@gmail.com>> wrote: I was under impression the “make test” was supposed to be passing when run under the regular user. That means that the recently added tap tests will make it fail: http://s5ci.myvpp.net/jobs/vpp-periodic-make-verify/ is only failures as of 12:25 UTC 24th December. So, what is the direction the community wants to go ? --a On 27 Dec 2019, at 17:22, Paul Vinciguerra <pvi...@vinciconsulting.com<mailto:pvi...@vinciconsulting.com>> wrote: I have been trying to figure out why I've been seeing different results between the CI jobs and s5ci jobs. What I've discovered is that the CI jobs are running as the root user in a fully privileged container, where s5ci runs unprivileged. I am aware that there has been a long-standing desire for tests to run as an unprivileged user. Do we need to implement that constraint on the CI tests? From my tests in the CI [0]: Sanity test case passed. Running as root(0):root(0) Capabilities: ['cap_chown', 'cap_dac_override', 'cap_dac_read_search', 'cap_fowner', 'cap_fsetid', 'cap_kill', 'cap_setgid', 'cap_setuid', 'cap_setpcap', 'cap_linux_immutable', 'cap_net_bind_service', 'cap_net_broadcast', 'cap_net_admin', 'cap_net_raw', 'cap_ipc_lock', 'cap_ipc_owner', 'cap_sys_module', 'cap_sys_rawio', 'cap_sys_chroot', 'cap_sys_ptrace', 'cap_sys_pacct', 'cap_sys_admin', 'cap_sys_boot', 'cap_sys_nice', 'cap_sys_resource', 'cap_sys_time', 'cap_sys_tty_config', 'cap_mknod', 'cap_lease', 'cap_audit_write', 'cap_audit_control', 'cap_setfcap', 'cap_mac_override', 'cap_mac_admin', 'cap_syslog', 'cap_wake_alarm', 'cap_block_suspend', 'cap_audit_read+eip'] OS reports 72 available cpu(s). Free shm: 1,023.9921875MB Found enough resources to run tests with 4 cores [0] https://logs.fd.io/production/vex-yul-rot-jenkins-1/vpp-verify-master-ubuntu1804/1331/console.log.gz -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#14981): https://lists.fd.io/g/vpp-dev/message/14981 Mute This Topic: https://lists.fd.io/mt/69288694/675608 Group Owner: vpp-dev+ow...@lists.fd.io<mailto:vpp-dev%2bow...@lists.fd.io> Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [ayour...@gmail.com<mailto:ayour...@gmail.com>] -=-=-=-=-=-=-=-=-=-=-=- -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#14990): https://lists.fd.io/g/vpp-dev/message/14990 Mute This Topic: https://lists.fd.io/mt/69288694/675642 Group Owner: vpp-dev+ow...@lists.fd.io<mailto:vpp-dev+ow...@lists.fd.io> Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [dmar...@me.com<mailto:dmar...@me.com>] -=-=-=-=-=-=-=-=-=-=-=-
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#14992): https://lists.fd.io/g/vpp-dev/message/14992 Mute This Topic: https://lists.fd.io/mt/69288694/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
