On 10/18/2013 12:04 PM, Brant Knudson wrote:
To provide a bit more background... Keystone has a bunch of
keystoneclient tests for the v2 API. These tests actually "git clone"
a version of keystoneclient (master, essex-3, and 0.1.1)[0], to use
for testing. Maybe at some point the tests were just for
client-server compatibility, but now they're used for more than that;
for example, they're used for tests that require going through the
paste pipeline and v2 controller. It's just the quickest way to get
some tests written. In addition, there's versions of the client tests
for both the kvs and sql backends.
This causes several problems,
1) It looks like we're not keeping the versions to test up-to-date --
should we checkout supported releases instead?
2) "git clone"ing the keystoneclient doesn't work well with parallel
testing (we have a similar problem in our tests with our "pristine"
database backup)
3) These tests eat up lots of memory which we've gotten complaints about.
Getting v3 API keystoneclient/keystone testing in tempest is going to
hopefully lead to getting the v2 tests out of Keystone. Thanks to
Steve for taking this first step!
I was going to try an approach where we used tempest to just call the
code in Keystone as a first step. That was one of the reasons that I
was in favor of moving the Keystone tests into the keystone namespace.
We need to skip the git clone, step, obviously, and I am not certain
about our use of fixtures: we might need to redo these so the sample
data doesn't conflict with what Tempest expects.
For the v3 API, the tests don't use the keystoneclient but instead use
webtest [1] and the REST API.
[0]
https://github.com/openstack/keystone/blob/master/keystone/tests/test_keystoneclient.py#L1070
[1]
https://github.com/openstack/keystone/blob/master/keystone/tests/test_content_types.py#L69
We'll need V3 client support eventually, and we should use Tempest as
the primary test environement for that.
- Brant
On Fri, Oct 18, 2013 at 10:59 AM, Dolph Mathews
<dolph.math...@gmail.com <mailto:dolph.math...@gmail.com>> wrote:
On Fri, Oct 18, 2013 at 10:34 AM, Steven Hardy <sha...@redhat.com
<mailto:sha...@redhat.com>> wrote:
Hi all,
Starting a thread to discuss $subject, as requested in:
https://review.openstack.org/#/c/51558/
First a bit of background. I wrote a keystoneclient patch,
and ayoung
stated he'd like it tested via tempest before he'd ack it:
https://review.openstack.org/#/c/48462/
So I spoke to ayoung and dkranz on IRC, showing them my local
tests for the
patch. dkranz suggested creating a "client_lib" directory,
where we could
build out a more comprehensive set of tests over time, adding
to the inital
tests related to keystone trusts client additions.
A couple of things to note:
- These are end-to-end tests, designed to test not only the
client, but
also the API and keystone backend functionality. So
arguably this could
just be a scenario test, e.g scenario/keystone/test_v3_auth.py
I'd love to be able to run these tests against a wider variety of
service configurations (e.g. LDAP!), which tempest is obviously
more suitable for.
- The intention is to excercise logic which is hard to fully
test with
unit or integration tests, and to catch issues like
incompatibility
between client and API - e.g keystoneclient tests may pass,
but we need
to make sure the client actually works against the real
keystone API.
All of our tests under keystone.tests.test_keystoneclient fall
into this category as well:
https://github.com/openstack/keystone/blob/a0e26c1882d83989bee3726a5ae08cbe3f32a2b5/keystone/tests/test_keystoneclient.py
https://github.com/openstack/keystone/blob/a0e26c1882d83989bee3726a5ae08cbe3f32a2b5/keystone/tests/test_keystoneclient_sql.py
Working on Heat has given me a pretty good insight into the
python-*client
API's, as we use them to orchestrate actions with every
openstack service;
IMO anything we can do to make these interfaces more robust
(and catch
bugs, several of which I found already while writing these
tests) is a
good-thing (tm).
++
I'd welcome feedback on the patch above, and what will be the most
acceptable approach to the tempest team for adding these tests.
More links:
https://review.openstack.org/#/c/51559/
https://review.openstack.org/#/c/51560/
https://blueprints.launchpad.net/tempest/+spec/keystoneclient-api
Thanks!
Steve
_______________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
<mailto:OpenStack-dev@lists.openstack.org>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
--
-Dolph
_______________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
<mailto:OpenStack-dev@lists.openstack.org>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
_______________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
_______________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
