Hi folks! Thanks all for the feedback. Yeah, this definitely needs more discussions, and the Summit is good place to have them. Thanks to Serg for offering a spot in the schedule. I hope to have some PoC by that time, so we can have something which can be demoed.
I'll try to attend the AppCatalog sessions as well. I didn't finalize my summit schedule yet, but I'll do my best to attend as much as I can. Thanks! -- Regards, Alexander Tivelkov On Fri, Oct 16, 2015 at 10:07 PM, Christopher Aedo <d...@aedo.net> wrote: > On Thu, Oct 15, 2015 at 3:04 AM, Alexander Tivelkov > <ativel...@mirantis.com> wrote: > > Hi folks, > > > > I’ve noticed that the Community Application Catalog has begun to > implement > > its own API, and it seems to me that we are going to have some > significant > > duplication of efforts with the code which has already been implemented > in > > Glance as Artifact Repository initiative (also known as Glance V3). > > From the very beginning of the App Catalog project (and I’ve been > involved > > in it since February) I’ve been proposing to use Glance as its backend, > > because from my point of view it covers like 90% of the needed > > functionality. But it looks like we have some kind of miscommunication > here, > > as I am getting some confusing questions and assumptions, like the > vision of > > Glance V3 being dedicated backend for Murano (which is definitely > > incorrect). > > So, I am writing the email to clarify my vision of what Glance V3 is and > how > > its features may be used to provide the REST API for Community App > Catalog. > > > > 1. Versioned schema > > First of all, Glance V3 operates on entities called “artifacts”, and > these > > ones perfectly map to the Data Assets of community app catalog. The > > artifacts are strongly typed: there are artifact types for murano > packages, > > glance images, heat templates - and there may be (and will be) more. Each > > artifact type is represented by a plugin, defining the schema of objects’ > > data and metadata and - optionally - custom logic. So, this thing is > > extensible: when a new type of asset needs to be added to a catalog it > can > > be done really quickly by just defining the schema and putting that > schema > > into a plugin. Also, these plugins are versioned, so the possible > changes in > > the schema are handled properly. > > > > 2. Generic properties > > Next, all the artifact types in Glance V3 have some generic metadata > > properties (i.e. part of the schema which is common for all the types), > > including the name, the version, description, authorship information and > so > > on. This also corresponds to the data schema of community app catalog. > The > > mapping is not 1:1, but we can work together on this to make sure that > these > > generic properties match the expectations of the catalog. > > > > 3. Versioning > > Versions are very important for catalogs of objects, so Glance V3 was > > initially designed keeping versioning questions in mind: each artifact > has a > > semver-based version assigned, so the artifacts having the same name but > > different versions are grouped into the proper sequences. API is able to > > query artifacts based on their version spec, e.g. it is possible to fetch > > the latest artifact with the name “foo” having the version greater than > 2.1 > > and less than 3.5.7 - or any other version spec, similar to pip or any > other > > similar tool. As far as I know, community app catalog does not have such > > capabilities right now - and I strongly believe that it is really a must > > have feature for a catalog to be successful. At least it is absolutely > > mandatory for Murano packages, which are the only “real apps” among the > > asset types right now. > > > > 4. Cross artifact dependencies > > Glance V3 also has the dependency relations from the very beginning, so > they > > may be defined as part of artifact type schema. As a result, an artifact > may > > “reference” any number of other artifacts with various semantic. For > > example, murano package may define a set of references to other murano > > packages and call it “requires” - and this will act similar to the > > requirements of a python package. Similar properties may be defined for > heat > > templates and glance images - they may reference each other with various > > semantics. > > Of course, the definitions of such dependencies may be done internally > > inside the packages, so they may be resolved locally by the service > which is > > going to use it, but letting the catalog know about them will allow us > to do > > the import-export operations for any given artifacts and its dependencies > > automatically, only by the means of the catalog itself. > > > > 5. Search and filtering API > > Right now Glance V3 API is in experimental state (we plan to stabilize it > > during the Mitaka cycle), but it already provides quite good > capabilities to > > discover things. It can search artifacts by their type, name and > > (optionally) aforementioned version specs, by tag or even by arbitrary > set > > of metadata properties. We have plans to integrate Glance V3 with the > > Searchlight project to have even more index and search capabilities using > > its elastic search engine. > > > > 6. Data storage > > As you probably know, Glance does not own the binary data of its images. > > Instead, it provides an abstraction of the backend storage, which may be > > swift, ceph, s3 or something else. The same approach is used in Glance V3 > > for artifacts data, but with more per-type control: particular artifact > > types may be configured independently to store their blobs in different > > backends. This may be of use for Community App Catalog which operates on > > different storages for its assets. > > > > 7. Sharing and access control. > > Glance V3 inherits the same access mechanics present in Glance V2: an > > artifact may be visible to its owner tenant only, be public (i.e. > visible to > > all the tenants) or directly shared by the owner to a specific tenant. > Also, > > Glance can act in the anonymous mode (i.e. without an access token), thus > > providing access to public artifacts even to unauthenticated users. > > This can be easily applied to a public web service, such as community app > > catalog: regular unauthenticated users use anonymous mode to access only > the > > public assets (this is the current behavior of apps.o.o), while > registered > > users will have their own private spaces (“tenants”) with various access > > restrictions. > > > > 8. The federation. > > The ultimate goal for Glance Artifact Repository is ability to build > trees > > of artifact repos in different clouds. The top node of that tree is some > > Global Application Catalog (and the apps.openstack.org may be this > global > > catalog - if it is glance-based or at least supports glance v3 > federation), > > then there are repositories of particular openstack vendors or > distributors, > > then - the catalogs of enterprises operating different openstack clouds. > The > > particular glance deployments in that clouds are the leafs of that tree, > > being able to search for data assets in all the upstream repositories, > > download them from there or - if permitted - submit their local assets > back > > upstream. This will be the ultimate network for application delivery and > > exchange in openstack world - and this is one of the main reasons we’ve > > began the Artifacts initiative in Glance. > > Unlike other aforementioned features this one is not implemented yet, > but we > > are planning to add it as soon as we are done with API stabilization > goal. > > > > > > There are many other features which are present in V3’s roadmap and may > be > > useful for the app catalog, such as ability to sign artifacts with their > > developers’ keys and verify that keys on usage to ensure the > authenticity of > > the artifact. > > > > What we don’t have right now is the ability to associate ratings > (“stars”) > > and comments to the artifact, as well as aggregating different usage and > > download statistics: such features are really needed only for the public > > website such as apps.o.o but are not required for Glance’s in particular > > clouds. But we may find some way to solve this, either by wrapping glance > > API with additional middleware which would add appropriate info from a > > different data source, or by having custom plugins which are able to do > > that, or in some other way: I am sure we may find a solution for this. > > > > So, this was just a brief description of what Glance v3 has to offer as a > > backend for App Catalog API. > > It also worths to mention that this API is in “EXPERIMENTAL” state right > > now, which means that it is not fixed and we may modify it significantly > if > > there is a need to. So we may work closer together to adopt it for the > needs > > of Community App Catalog. > > > > I would really prefer to not create any overlaps between Glance v3 and > the > > community app catalog: if the app catalog builds its own incompatible > > implementation of assets discovery and distribution API then we’ll have a > > huge duplication of efforts for developers and lots of confusion to the > > end-users who will get two entirely different ways to do the same task. > > > > So, I’d propose to discuss these potential overlaps, look at the features > > need by App Catalog and see how Glance V3 may be of use here. I’ll be > more > > than happy to help with that. We can dive deeper into the details here in > > the mailing list or meet in person in Tokyo. I'll try to have a > > demonstratable prototype by that time. > > Alexander, you make some great points here about the feature overlaps > between the Glance v3 API and where we are heading with the App > Catalog. There's good potential to re-use some of this work for sure, > and I'm really looking forward to what comes of the PoC you talked > about. > > Two important considerations against adopting Glance v3 wholesale > (aside from the experimental nature and the possibility that key bits > won't be ultimately adopted in the finished stable product) are > development velocity and deployment goals. As glance is a key > component in any OpenStack cloud, and is expected to be deployed in > hundreds (or thousands!) of environments, the dependencies and default > requirements are tightly coupled to most other OpenStack projects. > Obviously any changes to glance are weighed against how those will > impact all these clouds, how upgrades will work, backwards and > forwards compatibility, etc. For all the benefits that come with > working on a distributed open source project like this, it also tends > to mean we can't always move very fast because of the potential impact > across the landscape of the whole project. > > The App Catalog is in a different position - to put it really simply, > we're developing a web site intended to benefit user of OpenStack > clouds, and including some integration points to make it much easier > for users to find and share things that run on OpenStack clouds. But > we're not tightly coupled to any of the other projects, and we're not > expecting this to be something that's deployed on it's own with every > OpenStack cloud. In fact I'd argue deploying a local/private App > Catalog dilutes the value and harms the ecosystem as a whole. > > Ideally we'll find a balanced way to benefit from the great work > coming from the Glance team without limiting our design considerations > or encumbering the App Catalog with requirements or limitations that > shouldn't impact us. > > I'm really looking forward to speaking more about this in person in > Tokyo, hopefully you can make it to the App Catalog sessions! > > -Christopher > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
