On Fri, Sep 5, 2014 at 2:40 AM, Jay Pipes <jaypi...@gmail.com> wrote:
> On 09/04/2014 11:32 AM, Steven Hardy wrote: > >> On Thu, Sep 04, 2014 at 10:45:59AM -0400, Jay Pipes wrote: >> >>> On 08/29/2014 05:15 PM, Zane Bitter wrote: >>> >>>> On 29/08/14 14:27, Jay Pipes wrote: >>>> >>>>> On 08/26/2014 10:14 AM, Zane Bitter wrote: >>>>> >>>>>> Steve Baker has started the process of moving Heat tests out of the >>>>>> Tempest repository and into the Heat repository, and we're looking for >>>>>> some guidance on how they should be packaged in a consistent way. >>>>>> Apparently there are a few projects already packaging functional tests >>>>>> in the package <projectname>.tests.functional (alongside >>>>>> <projectname>.tests.unit for the unit tests). >>>>>> >>>>>> That strikes me as odd in our context, because while the unit tests >>>>>> run >>>>>> against the code in the package in which they are embedded, the >>>>>> functional tests run against some entirely different code - whatever >>>>>> OpenStack cloud you give it the auth URL and credentials for. So these >>>>>> tests run from the outside, just like their ancestors in Tempest do. >>>>>> >>>>>> There's all kinds of potential confusion here for users and packagers. >>>>>> None of it is fatal and all of it can be worked around, but if we >>>>>> refrain from doing the thing that makes zero conceptual sense then >>>>>> there >>>>>> will be no problem to work around :) >>>>>> >>>>>> I suspect from reading the previous thread about "In-tree functional >>>>>> test vision" that we may actually be dealing with three categories of >>>>>> test here rather than two: >>>>>> >>>>>> * Unit tests that run against the package they are embedded in >>>>>> * Functional tests that run against the package they are embedded in >>>>>> * Integration tests that run against a specified cloud >>>>>> >>>>>> i.e. the tests we are now trying to add to Heat might be qualitatively >>>>>> different from the <projectname>.tests.functional suites that already >>>>>> exist in a few projects. Perhaps someone from Neutron and/or Swift can >>>>>> confirm? >>>>>> >>>>>> I'd like to propose that tests of the third type get their own >>>>>> top-level >>>>>> package with a name of the form <projectname>-integrationtests (second >>>>>> choice: <projectname>-tempest on the principle that they're >>>>>> essentially >>>>>> plugins for Tempest). How would people feel about standardising that >>>>>> across OpenStack? >>>>>> >>>>> >>>>> By its nature, Heat is one of the only projects that would have >>>>> integration tests of this nature. For Nova, there are some "functional" >>>>> tests in nova/tests/integrated/ (yeah, badly named, I know) that are >>>>> tests of the REST API endpoints and running service daemons (the things >>>>> that are RPC endpoints), with a bunch of stuff faked out (like RPC >>>>> comms, image services, authentication and the hypervisor layer itself). >>>>> So, the "integrated" tests in Nova are really not testing integration >>>>> with other projects, but rather integration of the subsystems and >>>>> processes inside Nova. >>>>> >>>>> I'd support a policy that true integration tests -- tests that test the >>>>> interaction between multiple real OpenStack service endpoints -- be >>>>> left >>>>> entirely to Tempest. Functional tests that test interaction between >>>>> internal daemons and processes to a project should go into >>>>> /$project/tests/functional/. >>>>> >>>>> For Heat, I believe tests that rely on faked-out other OpenStack >>>>> services but stress the interaction between internal Heat >>>>> daemons/processes should be in /heat/tests/functional/ and any tests >>>>> the >>>>> rely on working, real OpenStack service endpoints should be in Tempest. >>>>> >>>> >>>> Well, the problem with that is that last time I checked there was >>>> exactly one Heat scenario test in Tempest because tempest-core doesn't >>>> have the bandwidth to merge all (any?) of the other ones folks >>>> submitted. >>>> >>>> So we're moving them to openstack/heat for the pure practical reason >>>> that it's the only way to get test coverage at all, rather than concerns >>>> about overloading the gate or theories about the best venue for >>>> cross-project integration testing. >>>> >>> >>> Hmm, speaking of passive aggressivity... >>> >>> Where can I see a discussion of the Heat integration tests with Tempest >>> QA >>> folks? If you give me some background on what efforts have been made >>> already >>> and what is remaining to be reviewed/merged/worked on, then I can try to >>> get >>> some resources dedicated to helping here. >>> >> >> We recieved some fairly strong criticism from sdague[1] earlier this year, >> at which point we were already actively working on improving test >> coverage >> by writing new tests for tempest. >> >> Since then, several folks, myself included, commited very significant >> amounts of additional effort to writing more tests for tempest, with some >> success. >> >> Ultimately the review latency and overhead involved in constantly rebasing >> changes between infrequent reviews has resulted in slow progress and >> significant frustration for those attempting to contribute new test cases. >> >> It's been clear for a while that tempest-core have significant bandwidth >> issues, as well as not necessarily always having the specific domain >> expertise to thoroughly review some tests related to project-specific >> behavior or functionality. >> >> So it was with some relief that we saw the proposal[2] to move the burden >> for reviewing project test-cases to the project teams, who will presumably >> be more motivated to do the reviews, and have the knowledge of what needs >> testing. >> >> [1] http://lists.openstack.org/pipermail/openstack-dev/2014- >> March/029661.html >> [2] http://lists.openstack.org/pipermail/openstack-dev/2014- >> July/041057.html >> >> I would greatly prefer just having a single source of integration >>> testing in >>> OpenStack, versus going back to the bad ol' days of everybody under the >>> sun >>> rewriting their own. >>> >>> Note that I'm not talking about functional testing here, just the >>> integration testing... >>> >> >> You may have to define the terms functional and integration here, as IMO >> there's already significant confusion about what the target of e.g API and >> scenario tests in tempest are. >> > > Yes, I already did this in a previous (snipped) portion of this email > thread. See here: > > http://lists.openstack.org/pipermail/openstack-dev/2014-August/044514.html > > yay for multi-month-crossing ML threads ;) > > > This is also further complicated by the fact that all heat functional >> tests >> also test integration of the various underlying services to some extent. >> > > Yes, as acknowledged in above post -- but Heat is special in this regard, > as mentioned in the post above. Functional tests for Heat are not the same > as functional tests for Nova, Keystone, Glance, Swift, etc. > > > My opinion is that any tests remaining in tempest should focus on API >> correctness, e.g to keep us honest in terms of backwards incomaptible >> changes to the API surface. >> > > OK, that's a perfectly reasonable suggestion. Would the Heat > functional^Wintegration tests then only test the latest version of the APIs? > > > Then for all tests which aim to prove the functionality of the project, >> e.g >> my understanding of tempest scenario tests atm, we should allow project >> teams to own them, and add to them as functionality develops over time. >> > > Well, that's where things get interesting. Functional tests in Glance and > Nova, for instance, are functional tests, not integration tests, because > they specifically do NOT test things like authentication -- i.e. the > interaction between Nova/Glance and Keystone. The Nova functional tests > likewise do not test the interaction between Nova and Glance, nor Nova and > the underlying hypervisor, as those layers are both faked out in the > functional Nova tests. > > As I mentioned in the above post, though, Heat is different. In order for > Heat to really have a worthwhile functional test, it would have to fake out > every single OpenStack service, since Heat itself does little more than > provide an This might be possible:-O Check out: https://openstacksummitnovember2014paris.sched.org/event/78969d086807fb8f60b29142149d9163#.VAka3da5s1w https://github.com/rackerlabs/mimic It would be neat to have a "fakestack" that heat (and others) could use. Then we run our functional tests against fakestack and the integration tests run the same tests just with a real devstack backing it. -Angus > orchestration REST API and template format that itself calls other > OpenStack services. And it is for this reason that I believe functional > tests for Heat belong in Tempest, since Tempest tests assume a working > OpenStack environment already, and there's no reason to either fake out all > the OpenStack services, nor any reason to develop a separate integration > testing framework inside of Heat that does the same things that Tempest > does. > > For Glance functional tests, they are testing the communication channels > between the Glance API service and the Glance registry service. For Nova > functional tests, they are testing the communication channels between the > Nova API, Nova conductor, Nova scheduler and Nova compute services. > > Heat does not have these internal interfaces to test communications for, > and this is why I don't think an in-Heat-tree functional test suite is > worth bothering with versus the (yes, frustrating) process of adding > tempest Heat tests. > > Best, > -jay > > > Ultimately I don't think it really matters which repo those tests live in, >> provided we can write them and get them running in the gate (catching >> regressions, which otherwise keep slipping through) in a timely manner. >> >> Steve >> >> _______________________________________________ >> OpenStack-dev mailing list >> OpenStack-dev@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> > _______________________________________________ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >
_______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
