rage-quit On Fri, Aug 4, 2017 at 11:52 AM, Monty Taylor <mord...@inaugust.com> wrote:
> On 08/04/2017 03:24 AM, Thierry Carrez wrote: > >> Michael Johnson wrote: >> >>> I was wondering what is the current status of the python-openstacksdk >>> project. The Octavia team has posted some patches implementing our new >>> Octavia v2 API [1] in the SDK, but we have not had any reviews. I have >>> also >>> asked some questions in #openstack-sdks with no responses. >>> I see that there are some maintenance patches getting merged and a pypi >>> release was made 6/14/17 (though not through releases project). I'm not >>> seeing any mailing list traffic and the IRC meetings seem to have ended >>> in >>> 2016. >>> >>> With all the recent contributor changes, I want to make sure the project >>> isn't adrift in the sea of OpenStack before we continue to spend >>> development >>> time implementing the SDK for Octavia. We were also planning to use it as >>> the backing for our dashboard project. >>> >>> Since it's not in the governance projects list I couldn't determine who >>> the >>> PTL to ping would be, so I decided to ping the dev mailing list. >>> >>> My questions: >>> 1. Is this project abandoned? >>> 2. Is there a plan to make it an official project? >>> 3. Should we continue to develop for it? >>> >> >> Thanks for raising this. >> >> Beyond its limited activity, another issue is that it's not an official >> project while its name make it a "default choice" for a lot of users >> (hard to blame them for thinking that >> http://git.openstack.org/cgit/openstack/python-openstacksdk is not the >> official Python SDK for OpenStack, but I digress). So I agree that the >> situation should be clarified. >> >> I know that Monty has pretty strong feelings about this too, so I'll >> wait for him to comment. >> > > Oh boy. I'd kind of hoped we'd make it to the PTG before starting this > conversation ... guess not. :) > > Concerns > -------- > > I share the same concerns Thierry listed above. Specifically: > > * It is not an official project, but its name leads people to believe it's > the "right" thing to use if they want to talk to OpenStack clouds using > Python. > > * The core team is small to begin with, but recently got hit in a major > way by shifts in company priorities. > > I think we can all agree that those are concerns. > > Three additional points: > > * The OpenStack AppDev group and the various appdev hackathons use shade, > not openstacksdk. This is what we have people out in the world recommending > people use when they write code that consumes OpenStack APIs. The Interop > challenges at the Summits so far have all used Ansible's OpenStack modules, > which are based on shade, because they were the thing that works. > > * Both shade and python-openstackclient have investigated using > openstacksdk as their REST layer but were unable to because it puts some > abstractions in layers that make it impossible to do some of the advanced > things we need. > > * openstacksdk has internal implementations of things that exist at > different points in the stack. We just added full support for version > service and version discovery to keystoneauth, but openstacksdk has its own > layer for that so it both can't use the ksa implementation and is not > compliant with the API-WG consume guidelines. > > It's not all bad! There is some **great** work in openstacksdk and it's a > shame there are some things that make it hard to consume. Brian, Qiming and > Terry have done a bunch of excellent work - and I'd like to not lose it to > the dustbin of corporate shifting interest. > > **warning** - there is a very large text wall that follows. If you don't > care a ton on this topic, please stop reading now, otherwise you might > rage-quit computers altogether. > > Proposal > -------- > > I'd propose we have the shade team adopt the python-openstacksdk codebase. > > This is obviously an aggressive suggestion and essentially represents a > takeover of a project. We don't have the luxury of humans to work on things > that we once had, so I think as a community we should be realistic about > the benefits of consolidation and the downside to continuing to have 2 > different python SDKs. > > Doing that implies the following: > > * Rework the underlying guts of openstacksdk to make it possible to > replace shade's REST layer with openstacksdk. openstacksdk still doesn't > have a 1.0 release, so we can break the few things we'll need to break. > > * Update the shade mission to indicate its purpose in life isn't just > hiding deployer differences but rather is to provide a holistic > cloud-centric (rather than service-centric) end-user API library. > > * Merge the two repos and retire one of them. Specifics on the mechanics > of this below, but this will either result in moving the resource and > service layer in openstacksdk into shade and adding appropriate attributes > to the shade.OpenStackCloud object, or moving the shade.OpenStackCloud into > something like openstack.cloud and making a shade backwards-compate shim. I > lean towards the first, as we've been telling devs "use shade to talk to > OpenStack" at hackathons and bootcamps and I'd rather avoid the messaging > shift. However, pointing to an SDK called "The Python OpenStack SDK" and > telling people to use it certainly has its benefits from a messaging > perspective. > > * Collapse the core teams - members of the python-openstacksdk-core team > who desire to stick around (I see Qiming doing reviews still, and Brian has > been doing occasional ones even after his day-job shift) are welcome to be > added to the shade-core team, but should not feel compelled to or like > they'd be letting anyone down if they didn't. Day job priorities shift, it > turns out. > > Reworking the Guts > ------------------ > > I did a scan through openstacksdk the other day to catalog what would need > to be reworked. The following are the big-ticket items: > > * drop stevedore/plugin support. An OpenStack REST client has no need for > plugins. All services are welcome. *note below* > > * drop keystoneauth.Session subclass. It's over-riding things at the wrong > layer. keystoneauth Adapter is the thing it wants to be. > > * stop using endpoint_filter in each Session call. Instead create an > Adapter with the discovery parameters needed. > > * add support for per-request microversions. Based on the structure > currently in openstacksdk once the wrapped Session has been replaced with > keystoneauth Adapter this should work really nicely. > > * drop incomplete version discovery support in favor of the support in > keystoneauth > > * drop Profile object completely and replace its use internally with the > os-client-config CloudConfig object. > > That's not a ton of work, TBH- I could probably do all of it in a single > long plane flight. It will break advanced users who might have been using > Profile (should be transparent to normal users) but as there is no 1.0 I > think we should live with that. We might be able to make a shim layer for > the Profile interface to avoid breaking people using the interface. > > *note on plugins* > > shade has a philosophy of not using plugins for service support that I'd > like to apply here. All OpenStack services are welcome to add code > directly. openstacksdk ALREADY contains code for tons of services. The only > thing plug-ability adds in this context is the ability to use openstacksdk > to support non-OpenStack services... and at this point I do not think that > is valuable. The only place this is currently used is in the Profile anyway > which allows defining an entrypoint to use to override a service - and > since I'm proposing we kill the Profile object this all falls out as a > matter of consequence. > > Integrating with shade > ---------------------- > > The primary end-user concept in shade is an OpenStackCloud object, on > which one performs actions. The service that provides the action is > abstracted (this is done because actions such as 'list_images' may need to > be done on the image service or the compute service, depending on the > cloud). So a user does: > > cloud = shade.openstack_cloud(cloud='example') > images = cloud.list_images() > > The primary end-user concept in openstacksdk is the Connection, which has > an object for each service. For example: > > conn = openstack.connection.from_config(cloud_name='example') > images = conn.image.images() > > If we merge the two with the shade library being the primary interface, we > could add the current sdk service proxy objects as attributes to the > OpenStackCloud object, so that the following would work: > > cloud = shade.openstack_cloud(cloud='example') > images = cloud.list_images() > images = cloud.image.images() > > If we did the merge the other way, we could either keep the Connection > concept and stitch the shade helpers on to it: > > conn = openstack.connection.from_config(cloud_name='example') > images = conn.list_images() > images = conn.image.images() > > Or do a hybrid: > > cloud = openstack.cloud(name='example') > images = cloud.list_images() > images = cloud.image.images() > > If we go either of the routes of merging shade into openstacksdk then the > shade library itself could just be a simple sugar layer for backwards > compat that has things like: > > def openstack_cloud(cloud=None, *args, **kwargs): > return openstack.cloud(name=cloud, *args, **kwargs) > > and > > class OpenStackCloud(openstack.cloud): > def __init__(self, cloud=None, *args, **kwargs): > super(OpenStackCloud, self).__init__(name=cloud, *args, **kwargs) > > I kind of like the 'Connection' term, as it communicates that this is a > thing that has and shares a discrete remote connection (which is a shared > keystoneauth Session) OpenStackCloud in shade **ACTUALLY** describes a > cloud-region (as regions in OpenStack are essentially independent clouds > from an API consumption perspective. So I may be leaning more towards > merging in that direction. > > * data model - shade has a data model contract for the resources its knows > about. This actually fits nicely with the Resource construct in > openstacksdk, although there are some differences. We should likely push > the data model and normalization contract into the openstacksdk resource > layer so that people get matching resources regardless of whether they use > the shade interop layer or the low-level per-service layer. > > * Implement some constructor smarts for easy pass-through of sdk service > proxy methods into shade wrapper methods. For MANY of the remote calls, the > list_ get_ search_ create_ update_ and delete_ methods are (or can be) > mechanical passthrough from the SDK objects. We should be able to write > some smart constructor logic that makes all the passthrough methods for us > and just have explicit methods defined for the places where the shade layer > legitimately needs to do a bunch of logic (like image upload and auto-ip > support) > > * Make openstack.resource2.Resource a subclass of munch.Munch. This will > be fun. If we want to be able to have model/normalization happen at the > lower-level, we'd ultimately want the shade methods to be able to just > return the object the sdk layer produces. Shade's interface defines that we > return Munch objects (basically things that behave like dicts and objects) > That's VERY similar to what the Resource already does - so if we subclass > Munch the shade behavior will hold and the sdk coding style should also be > able to hold as it is today. Otherwise we'd have to have every return in > shade wrap the object in a munch.Munch(resource.to_dict()) which would > lose information for when that object wants to be passed back in to the API > later. (there are smarts in Resource for doing update things) > > * Migrate openstacksdk unit tests to use requests_mock. We're pretty much > finished doing this in shade and it's been SUPER helpful. openstacksdk has > a mock of the Session object in test_proxy_base2.py, so there's some good > places to lay this in ... and as we merge the two obviously shade's > requests_mock unittests will continue to apply - but given the sdk's test > organization I think we can get some really solid results by moving the > mocking to the actual rest payload layer instead of mocking out the Session > itself. It also is a great way to verify that things work as expected with > varying payloads - so as a user finds an edge-case from a specific cloud, > grabbing an http trace from them and constructing a specific test is a > great way to deal with regressions. > > We'll also need to make a specific rollout plan. shade has a strict > backwards-compat policy, so if we merge sdk into shade, it goes from being > 0.9 to being fully supported very quickly and we need to make sure we don't > have anything exposed in a public interface we don't want to support for > ages (resource -> resource2 should likely get finished and then resource2 > should likely get renamed back to resource before such a release, for > instance) If we merge the other direction and make the current shade a > backwards-compat shim lib, we'll also need to cut a 1.0 of sdk pretty > quickly, as whatever passthrough object we are exposing via the shade layer > from sdk just adopted the shade backwards compat contract. I don't have a > detailed plan for this yet, but if we decide to go down this path I'll make > one. > > Other TODO list items > --------------------- > > It's not just all grunt work nobody can see. There's fun things to do too! > > * Add openstack.OpenStack or shade.AllClouds class - It's been a todo-list > item for me for a while to make a wrapper class in shade that allows easy > operations on a set of Clouds. Again, I like the semantics of > openstack.OpenStack better - so that's another argument in favor of that > direction of merge ... it would look something like this: > > # make Connection objects for every cloud-region in clouds.yaml > clouds = openstack.OpenStack() > # thread/asyncio parallel fetch of images across all clouds > images = clouds.list_images() > # all objects in shade have a "location" field which contains > # cloud, region, domain and project info for the resource > print([image for image in images if image.location.cloud='vexxhost']) > # Location is a required parameter for creation > vexxhost = clouds.get_location(name='vexxhost') > clouds.create_image( > location=vexxhost, name='my-fedora-26', > filename='fedora26.qcow2') > > Most of this work can actually likely be done with one smart metaclass ... > finding the methods on OpenStackCloud and either doing a set of parallel > gets on a list of objects or adding a location argument to write operations > doesn't vary depending on the type of object. Since all the methods are > named consistently, finding 'list_*' and making a set of corresponding > list_ methods on the OpenStack object is essentially just one chunk of > smart constructor. > > * Finish per-resource caching / batched client-side rate-limiting work. > shade has a crazy cool ability to do batched and rate-limited operations > ... this is how nodepool works at the scale it does. But it's currently > only really plumbed through for server, floating-ip and port. (guess what > nodepool has to deal with) This should be generalized to all of the > resources, configurable on a per-resource name in clouds.yaml, and should > work whether high or low level interfaces are used. This is super hard to > get RIGHT, so it's one of those "spend 4 weeks writing 10 lines of code" > kind of things, but it's also super important. > > * Implement a flag for toggling list/client-side-filter vs. remote-get > operations. Most single-resource operations in shade are actually done as a > list followed by a client-side filter. Again this model is there to support > nodepool-scale (amusingly enough it makes it less load on the clouds at > scale) but at small scale it is more costly and some users find it > annoying. We've discussed having the ability to toggle this at constructor > time - and then having things like the ansible modules default the flag to > use remote-get instead of list/filter - since those are doing lots of > independent processes so the optimization of list/filter isn't ever > realized. > > * Implement smarter and more comprehensive "pushdown" filtering. I think > we can piggyback a bunch of this off of the SDK layer - but there are > attributes that can be used for server-side filtering, there are attributes > that can't, and there are attributes that are client-side created via > normalization that either can be translated into a server-side filter or > must be client-side filtered. Resource has the structure for dealing with > this sanely I believe, but it needs to be tracked through. > > * Add python3-style type annotations. We just started doing this in zuul > and it's pretty cool - and is possible to do in a python2 compatible way. > > Longer Goals > ------------ > > That gets us a home for openstacksdk, a path towards consolidation of > effort and a clear story for our end users. There are a few longer-term > things we should be keeping in mind as we work on this: > > * suitability for python-openstackclient. Dean and Steve have been laying > in the groundwork for doing direct-REST in python-openstackclient because > python-*client are a mess from an end-user perspective and openstacksdk > isn't suitable. If we can sync on requirements hopefully we can produce > something that python-openstackclient can honestly use for that layer > instead of needing local code. > > * suitability for heat/horizon - both heat and horizon make calls to other > OpenStack services as a primary operation (plenty of services make > service-to-service calls, but for heat and horizon is a BIG part of their > life) The results of this work should allow heat and horizon to remove > local work they have using python-*client, doing local version discovery or > any of the rest - and should expose to them rich primitives they can use > easily. > > Conclusion > ---------- > > As I mentioned at the top, I'd been thinking some of this already and had > planned on chatting with folks in person at the PTG, but it seems we're at > a place where that's potentially counter productive. > > Depending on what people think I can follow this up with some governance > resolutions and more detailed specs. > > Thanks! > Monty > > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > -- Kind regards, Melvin Hillsman mrhills...@gmail.com mobile: (832) 264-2646
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
