On Thu, Sep 24, 2015 at 1:58 PM, Emilien Macchi <emil...@redhat.com> wrote: > > > On 09/24/2015 02:19 PM, Alex Schultz wrote: >> On Thu, Sep 24, 2015 at 11:54 AM, Emilien Macchi <emil...@redhat.com> wrote: >>> >>> >>> On 09/24/2015 10:14 AM, Alex Schultz wrote: >>>> On Wed, Sep 23, 2015 at 4:56 PM, Emilien Macchi <emil...@redhat.com> wrote: >>>>> Background >>>>> ========== >>>>> >>>>> Current rspec tests are tested with modules mentioned in .fixtures.yaml >>>>> file of each module. >>>>> >>>>> * the file is not consistent across all modules >>>>> * it hardcodes module names & versions >>>>> * this way does not allow to use "Depend-On" feature, that would allow >>>>> to test cross-modules patches >>>>> >>>>> Proposal >>>>> ======== >>>>> >>>>> * Like we do in beaker & integration jobs, use zuul-cloner to clone >>>>> modules in our CI jobs. >>>>> * Use r10k to prepare fixtures modules. >>>>> * Use Puppetfile hosted by openstack/puppet-openstack-integration >>>>> >>>>> In that way: >>>>> * we will have modules name + versions testing consistency across all >>>>> modules >>>>> * the same Puppetfile would be used by unit/beaker/integration testing. >>>>> * the patch that pass tests on your laptop would pass tests in upstream CI >>>>> * if you don't have zuul-cloner on your laptop, don't worry it will use >>>>> git clone. Though you won't have Depends-On feature working on your >>>>> laptop (technically not possible). >>>>> * Though your patch will support Depends-On in OpenStack Infra for unit >>>>> tests. If you submit a patch in puppet-openstacklib that drop something >>>>> wrong, you can send a patch in puppet-nova that will test it, and unit >>>>> tests will fail. >>>>> >>>>> Drawbacks >>>>> ========= >>>>> * cloning from .fixtures.yaml takes ~ 10 seconds >>>>> * using r10k + zuul-clone takes ~50 seconds (more modules to clone). >>>>> >>>>> I think 40 seconds is something accept regarding the benefit. >>>>> >>>> >>>> As someone who consumes these modules downstream and has our own CI >>>> setup to run the rspec items, this ties it too closely to the >>>> openstack infrastructure. If we replace the .fixtures.yml with >>>> zuul-cloner, it assumes I always want the openstack version of the >>>> modules. This is not necessarily true. I like being able to replace >>>> items within fixtures.yml when doing dev work. For example If i want >>>> to test upgrading another module not related to openstack, like >>>> inifile, how does that work with the proposed solution? This is also >>>> moving away from general puppet module conventions for testing. My >>>> preference would be that this be a different task and we have both >>>> .fixtures.yml (for general use/development) and the zuul method of >>>> cloning (for CI). You have to also think about this from a consumer >>>> standpoint and this is adding an external dependency on the OpenStack >>>> infrastructure for anyone trying to run rspec or trying to consume the >>>> published versions from the forge. Would I be able to run these tests >>>> in an offline mode with this change? With the .fixures.yml it's a >>>> minor edit to switch to local versions. Is the same true for the >>>> zuul-cloner version? >>> >>> What you did before: >>> * Edit .fixtures.yaml and put the version you like. >>> >>> What you would do this the current proposal: >>> * Edit openstack/puppet-openstack-integration/Puppetfile and put the >>> version you like. >>> >> >> So I have to edit a file in another module to test changes in >> puppet-neutron, puppet-nova, etc? With the zuul-cloner version, for >> local testing what does that workflow look like? > > If you need to test your code with cross-project dependencies, having > current .fixtures.yaml or the proposal won't change anything regarding > that, you'll still have to trick the YAML file that define the modules > name/versions. > >> >>> What you're suggesting has a huge downside: >>> People will still use fixtures by default and not test what is actually >>> tested by our CI. >>> A few people will know about the specific Rake task so a few people will >>> test exactly what upstream does. That will cause frustration to the most >>> of people who will see tests failing in our CI and not on their laptop. >>> I'm not sure we want that. >> >> You're right that the specific rake task may not be ideal. But that >> was one option, another option could be use fixtures first then >> replace with zuul-cloner provided versions but provide me the ability >> to turn of the zuul cloner part? I'm just saying as it is today, this >> change adds more complexity and hard ties into the OpenStack >> infrastructure with non-trival work arounds. I would love to solve the >> Depends-On issue, but I don't think that should include a deviation >> from generally accepted testing practices of puppet modules. > > I agree it's not best practice in Puppet but I don't see that as an huge > blocker. Our Puppet modules are Approved by Puppetlabs and respect most > of best practices AFIK. Is that fixctures thing a big deal? > I would like to hear from *cough*Hunner/Cody*cough* Puppetlabs about that. > Another proposal is welcome though, please go ahead. >
IMHO, it's more of a new developer thing. As a person who works on other puppet modules, to have a completely different dependency method for the OpenStack modules raises the barrier to entry for development. It could be addressed with better documentation... >>> >>> I think more than most of people that run tests on their laptops want to >>> see them passing in upstream CI. >>> The few people that want to trick versions & modules, will have to run >>> Rake, trick the Puppetfile and run Rake again. It's not a big deal and >>> I'm sure this few people can deal with that. >>> >> >> So for me the zuul-cloner task seems more of a CI specific job that >> solves the Depends-On issues we currently have. Much like the beaker >> and acceptance tests that's not something I run locally. > > Hum. We implemented beaker tests in our modules so you can test the > module on your infra (laptop/cloud/whatever). > Here, we're just talking about unit testing, but it's still testing > after all. > > Beaker code is already using this proposal to clone the module. > The proposal is re-using the same code to be consistent, and keep one > single centralized Puppetfile. > >> I usually run the local rspec tests first before shipping off to CI to see >> how that >> plays out but I would manage the .fixtures.yml if necessary to test >> cross module dependancies. I don't expect to replicate an entire CI >> environment setup on my laptop for testing. > > This proposal do not "replicate an entire CI". It just clone all > modules, on the same version, hence the 40 seconds difference. > The rspec tests will still work for you, and you won't see any difference. > >> The rspec tests for me, represent a quick way to test fixes before shipping >> off to CI for more >> testing. > > This is wrong. What you test on your laptop is not what we test in > upstream CI: not the same modules, not the same dependencies. > OpenStack Puppet modules are working with (upstream) dependencies that > help to build OpenStack Clouds. > > OpenStack is already running the same structure with Global Requirements > [1] (Python dependencies), where each project works with it. > > [1] http://git.openstack.org/cgit/openstack/requirements > > With this proposal, our modules will follow the same concept, where they > would be tested (unit + functional) against the same dependencies. > Sorry, by this I meant for additional version testing. Rather than running puppet 3.{4,5,6,7,8} and 4.x locally. Yes we need to test with the same version of the modules as would be in CI but that would be where the .fixtures.yml comes in. >> Going back to the background item from the original email, the >> .fixtures.yml shouldn't be identical for all modules. It should only >> be the modules required to test the specific module. I doubt all of >> the puppet OpenStack modules require each other, right? So that's not >> a problem, that's an expectation. Additionally, we should be managing >> these anyway so when we publish the modules to the forge, it has >> proper metadata indicating the dependancies. > > All modules have quite often openstacklib/mysql/rabbitmq/qpid/keystone > at least. > >> This change seems targeted towards solving OpenStack CI environmental >> setup issues, and not really improving individual module development >> from a regular puppet standpoint. > > Nope, if you read again the proposal, it's solving: > "I would like to run tests on my laptop that will pass OpenStack CI with > the right modules and the right versions". > > If you want to run your specific modules & version, I guess you'll have > to do like you're doing with fixtures: running rake (stop it after > cloning Puppetfile), editing Puppetfile, running rake again, instead of > just editing fixtures. > That is not very user-friendly and that's what I was referring to with the non-trival work arounds. This would make this a very OpenStack puppet specific development workflow. I also have concerns around the downstream CI implications for this. Being able to run tests without internet connectivity is important to some people so I want to make sure that can continue without having to break the process mid-cycle to try and inject a workaround. It would better if we could have a workaround upfront. For example make a Puppetfile location an environment variable and if not defined pull down the puppet-openstack-integration one? I wish there was a better dependency resolution method that just pulling everything down from the internets. I just know that doesn't work everywhere. >>>>> >>>>> Next steps >>>>> ========== >>>>> >>>>> * PoC in puppet-nova: https://review.openstack.org/#/c/226830/ >>>>> * Patch openstack/puppet-modulesync-config to be consistent across all >>>>> our modules. >>>>> >>>>> Bonus >>>>> ===== >>>>> we might need (asap) a canary job for puppet-openstack-integration >>>>> repository, that would run tests on a puppet-* module (since we're using >>>>> install_modules.sh & Puppetfile files in puppet-* modules). >>>>> Nothing has been done yet for this work. >>>>> >>>>> >>>>> Thoughts? >>>>> -- >>>>> Emilien Macchi >>>>> >>>>> >>>> >>>> I think we need this functionality, I just don't think it's a >>>> replacement for the .fixures.yml. >>>> >>>> Thanks, >>>> -Alex >>>> >>>> __________________________________________________________________________ >>>> OpenStack Development Mailing List (not for usage questions) >>>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>> >>> >>> -- >>> Emilien Macchi >>> >>> >>> __________________________________________________________________________ >>> OpenStack Development Mailing List (not for usage questions) >>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> >> >> __________________________________________________________________________ >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > > -- > Emilien Macchi > > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
