On Thu, Sep 29, 2016 at 5:27 AM, milanisko k <vetri...@gmail.com> wrote:
> > > út 27. 9. 2016 v 20:12 odesílatel Assaf Muller <as...@redhat.com> napsal: > >> On Tue, Sep 27, 2016 at 2:05 PM, Assaf Muller <as...@redhat.com> wrote: >> >>> >>> >>> On Tue, Sep 27, 2016 at 12:18 PM, Timur Nurlygayanov < >>> tnurlygaya...@mirantis.com> wrote: >>> >>>> Hi milan, >>>> >>>> we have measured the test coverage for OpenStack components with >>>> coverage.py tool [1]. It is very easy tool and it allows measure the >>>> coverage by lines of code and etc. (several metrics are available). >>>> >>>> [1] https://coverage.readthedocs.io/en/coverage-4.2/ >>>> >>> >>> coverage also supports aggregating results from multiple runs, so you >>> can measure results from combinations such as: >>> >>> >> >>> 1) Unit tests >>> 2) Functional tests >>> 3) Integration tests >>> 4) 1 + 2 >>> 5) 1 + 2 + 3 >>> >>> To my eyes 3 and 4 make the most sense. Unit and functional tests are >>> supposed to give you low level coverage, keeping in mind that 'functional >>> tests' is an overloaded term and actually means something else in every >>> community. Integration tests aren't about code coverage, they're about user >>> facing flows, so it'd be interesting to measure coverage >>> from integration tests, >>> >> >> Sorry, replace integration with unit + functional. >> >> >>> then comparing coverage coming from integration tests, and getting the >>> set difference between the two: That's the area that needs more unit and >>> functional tests. >>> >> >> To reiterate: >> >> Run coverage from integration tests, let this be c >> Run coverage from unit and functional tests, let this be c' >> >> Let diff = c \ c' >> >> 'diff' is where you're missing unit and functional tests coverage. >> > > Assaf, the tool I linked is a monkey-patched coverage.py but the collector > stores the stats in Redis --- gives the same accumulative collecting. > Is there any interest/effort to collect coverage stats from selected jobs > in CI, no matter the tool used? > Some projects already collect coverage stats on their post-merge queue: http://logs.openstack.org/61/61af70a734b99e61e751cfb494ddc93a85eec394/post/nova-coverage-db-ubuntu-xenial/55210aa/ It's invoked with 'tox -e cover' which you define in your project's tox.ini file, I imagine most projects if not all have it set up to gather coverage from a unit tests run. > > >> >> >>> >>> >>>> >>>> On Tue, Sep 27, 2016 at 1:06 PM, Jordan Pittier < >>>> jordan.pitt...@scality.com> wrote: >>>> >>>>> Hi, >>>>> >>>>> On Tue, Sep 27, 2016 at 11:43 AM, milanisko k <vetri...@gmail.com> >>>>> wrote: >>>>> >>>>>> Dear Stackers, >>>>>> I'd like to gather some overview on the $Sub: is there some >>>>>> infrastructure in place to gather such stats? Are there any groups >>>>>> interested in it? Any plans to establish such infrastructure? >>>>>> >>>>> I am working on such a tool with mixed results so far. Here's my >>>>> approach taking let's say Nova as an example: >>>>> >>>>> 1) Print all the routes known to nova (available as a python-routes >>>>> object: nova.api.openstack.compute.APIRouterV21()) >>>>> 2) "Normalize" the Nova routes >>>>> 3) Take the logs produced by Tempest during a tempest run (in >>>>> logs/tempest.txt.gz). Grep for what looks like a Nova URL (based on port >>>>> 8774) >>>>> 4) "Normalize" the tested-by-tempest Nova routes. >>>>> 5) Compare the two sets of routes >>>>> 6) ???? >>>>> 7) Profit !! >>>>> >>>>> So the hard part is obviously the normalizing of the URLs. I am >>>>> currently using a tons of regex.... :) That's not fun. >>>>> >>>>> I'll let you guys know if I have something to show. >>>>> >>>>> I think there's real interest on the topic (it comes up every year or >>>>> so), but no definitive answer/tool. >>>>> >>>>> Cheers, >>>>> Jordan >>>>> >>>>> >>>>> >>>>> >>>>> <https://www.scality.com/backup/?utm_source=signatures&utm_medium=email&utm_campaign=backup2016> >>>>> ____________________________________________________________ >>>>> ______________ >>>>> OpenStack Development Mailing List (not for usage questions) >>>>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject: >>>>> unsubscribe >>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>>> >>>>> >>>> >>>> >>>> -- >>>> >>>> Timur, >>>> Senior QA Manager >>>> OpenStack Projects >>>> Mirantis Inc >>>> >>>> ____________________________________________________________ >>>> ______________ >>>> OpenStack Development Mailing List (not for usage questions) >>>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject: >>>> unsubscribe >>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>> >>>> >>> ____________________________________________________________ >> ______________ >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject: >> unsubscribe >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
