On 10/22/2012 12:41 PM, Yaniv Kaul wrote: >> 1) Tempest executes a series of HTTP calls against public REST endpoints >> in OpenStack. It has no way of determining what code was run **on the >> server**. It only has the ability to know what Tempest itself executed, >> not what percentage of the total API those calls represented > > But you can get number of lines covered (vs. total number of lines in > the component).
No, you can't... you'd need to run this on the server side, and Tempest is a client-side thing -- you just point it at an OpenStack environment. You can't assume that you would be able to profile/SSH as root into the endpoint running the server to do the above. >> 2) Specs don't always exist for the APIs -- yes, I know, this isn't >> good. Especially problematic are some of the Compute API extensions that >> aren't documented well, or at all. > > which is why you need to look at the code, see what is missed (and > 'reverse-engineer' it into a test case). That's what we've been doing up until now. > But I guess it doesn't have to be either-or and could be both eventually > - I don't have a lot of faith in a wiki that needs updating, that's all. Well, like I mentioned in the original post, doing it in the wiki or automating/scripting some output is fine with me. I agree that a wiki is not ideal, but unless there is some code-readable description of the entire API -- via XSchema/JSONSchema, etc -- this isn't very easy. Best, -jay > Y. > >> >> Best, >> -jay >> >>> Thanks, >>> Y. >>> >>>> * I will start the traceability matrix stuff and publish for people to >>>> go and update >>>> * Antoni from HP is going to investigate using things in testtools and >>>> testrepository for handling module and package-level fixtures and >>>> removing some of the nose-based cruft >>>> * I will be working on the YAML file that describes a test environment >>>> so that different teams that use different deployment frameworks can use >>>> a commonly-agreed format to describe their envs to the CI system >>>> * A new member of Gigi's team (really sorry, I've forgotten your name! >>>> :( ) is going to look at the fuzz testing discussion from earlier and >>>> see about prototyping something together that would be used for negative >>>> and security/fuzz testing -- this would enable us to remove the static >>>> negative tests from Tempest's main test directories. For the record (and >>>> for the team member whose name I have forgotten, here is the relevant >>>> link: https://lists.launchpad.net/openstack-qa-team/msg00155.html and >>>> https://lists.launchpad.net/openstack-qa-team/msg00156.html) >>>> >>>> Best, >>>> -jay >>>> >>>> On 10/19/2012 02:47 PM, Sean Gallagher wrote: >>>>> Daryl, >>>>> I wasn't able to make the follow-up meeting. :/ >>>>> >>>>> Can you/ someone send out a recap? >>>>> >>>>> David: you had a list of items. Can you post or share that somewhere? >>>>> >>>>> We discussed Blueprints for managing some of the planning work. >>>>> >>>>> What about higher level planning docs? Useful? Do we have any? Should we? >>>>> >>>>> Re Google Hangout next week, I'm interested. >>>>> >>>>> -sean >>>>> > -- Mailing list: https://launchpad.net/~openstack-qa-team Post to : openstack-qa-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack-qa-team More help : https://help.launchpad.net/ListHelp
