Hi Darragh, SimpleCov is a ruby-only solution and the puppet code (*.pp) coverage from rspec-puppet does not integrate into it. As far as I can see there is currently no way to get at the raw data without hacking rspec-puppet itself <https://github.com/rodjek/rspec-puppet/blob/3a053673a32587e3507597a7527e1a90a8425cc4/lib/rspec-puppet/coverage.rb#L72> .
Cheers, David On Fri, Jun 15, 2018 at 4:04 PM Darragh Bailey <daragh.bai...@gmail.com> wrote: > Hi, > > > I'm to understand how to get a full coverage for puppet code with spec > tests to be generated so I can get output like the example from simplecov > at https://github.com/colszowka/simplecov#example-output. > > Ideally I would subsequently try to hook into the coverage repo and do > some post processing looking at where most of the code churn has been in > our puppet control code repo and target those areas as important for test > coverage (and likely refactoring). > > However I've stuck on not being able to get output from simplecov that > shows the manifests that were tested by the spec tests, although the > summary triggered from running 'RSpec::Puppet::Coverage.report!' suggests > that something was capable of tracking coverage correctly. > > Although we have a different layout than normal, my testing so far > suggests that the reporting simply doesn't work between puppet and > simplecov. > > Usually when something like this happens, I assume I'm doing something > blindingly stupid, so I've put together some example code that shows my > problem should anyone like to run through the spec tests from it. > > Doing some simple experiments I've pushed up a repository showing the > issue https://github.com/electrofelix/puppet-simplecov_bug containing 3 > commits > - (HEAD commit): convert to match more closely skeleton layout from > https://github.com/garethr/puppet-module-skeleton/blob/master/skeleton/Gemfile > - (HEAD~1 commit): try to follow a more standard layout > - (root commit): Use simple define taken from our codebase and matching > the layout we have as closely as possible > > > Note we're still stuck on Puppet 3.8 and using ruby 2.1.10 via rvm while I > was doing the spec tests, plan to upgrade to 5 in the near future, so if > this problem has been solved via something in the puppet engine (some > suggestions that the multiple loading by puppet was wiping SimpleCov's > tracking over coverage) and we just need to hurry up that's great. > > But no matter what layout/format I seem to try, the simplecov report > prints nothing as having been covered, e.g.: > > > *************************************************************************************************** > Total resources: 9 > Touched resources: 8 > Resource coverage: 88.89% > Untouched resources: > > Ssh::Config_entry[example custom-host] > > Finished in 2.01 seconds (files took 1.05 seconds to load) > 12 examples, 0 failures > Coverage report generated for RSpec to <path>/puppet-simplecov_bug/html. 0 > / 67 LOC (0.0%) covered. > > COVERAGE: 0.00% -- 0/67 lines in 2 files > > > +----------+-------------------------------+-------+--------+-----------------------------------------+ > | coverage | file | lines | missed | > missing | > > +----------+-------------------------------+-------+--------+-----------------------------------------+ > | 0.00% | manifests/server/localuser.pp | 67 | 67 | 5-13, 15-19, > 21-34, 36-44, 46-55, 57-76 | > > +----------+-------------------------------+-------+--------+-----------------------------------------+ > 1 file(s) with 100% coverage not shown > > *************************************************************************************************** > > RSpec::Puppet::Coverage.report causes the first part to be printed and > certainly it shows it's tracking coverage on the files touched, however the > simplecov report at the end does not appear to be able to correlate the > tests in 'spec/defines/server_localuser_spec.rb' as having tested > 'manifests/server/localuser.pp'. > > Additionally what is missing from both is all files that are not touched > at all so far by tests, though I do have some code that appears to be able > to handle that by passing 'track_files("**/*.pp")' to the SimpleCov object > configuration. > > > > Thinking maybe I started from the wrong place, I took > https://github.com/garethr/puppet-module-skeleton and created a fresh > module from that and then dropped in both the spec file ( > https://github.com/electrofelix/puppet-simplecov_bug/blob/master/spec/defines/server_localuser_spec.rb) > and define ( > https://github.com/electrofelix/puppet-simplecov_bug/blob/master/manifests/server/localuser.pp) > that were being tested and the result remained the same. > > > Rspec::Puppet::Coverage.report! gave: > > Total resources: 15 > Touched resources: 14 > Resource coverage: 93.33% > Untouched resources: > > Ssh::Config_entry[example custom-host] > > > But SimpleCov gave: > > Coverage report generated for RSpec to <path>/test-coverage/coverage. 0.0 > / 0.0 LOC (100.0%) covered. > > > So even worse than before... > > > Resources I've looked at: > > https://github.com/garethr/puppet-module-skeleton > https://github.com/rodjek/rspec-puppet > https://github.com/rodjek/rspec-puppet#producing-coverage-reports > https://github.com/rodjek/rspec-puppet/issues/316 > https://github.com/puppetlabs/puppetlabs-rook (seems to have the same > issue) > > https://github.com/colszowka/simplecov#example-output > https://github.com/colszowka/simplecov/issues/682 > https://github.com/colszowka/simplecov/issues/389 > https://github.com/colszowka/simplecov/issues/340 > https://github.com/colszowka/simplecov/issues/77 > > my example of the issue: > https://github.com/electrofelix/puppet-simplecov_bug > > > Hoping that there is something relatively trivial here that I'm missing > that allows the reports generated by simplecov to correctly reference spec > tests with puppet code. > > > > Regards, > Darragh > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to puppet-users+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/puppet-users/3afe6401-44d9-4bcb-8628-6bc76e4b3491%40googlegroups.com > <https://groups.google.com/d/msgid/puppet-users/3afe6401-44d9-4bcb-8628-6bc76e4b3491%40googlegroups.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- Cheers, David https://twitter.com/dev_el_ops -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CALF7fHao7EMwGGA6KLDfuGPtBh%3D3GPbwte%3Dmi7fpaUgpUUWVmg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
