I see that the new client plugins are loaded using stevedore, which is
great and IMO absolutely the right tool for that job. Thanks to Angus &
Steve B for implementing it.
Now that we have done that work, I think there are more places we can
take advantage of it too - for example, we currently have competing
native wait condition resource types being implemented by Jason[1] and
Steve H[2] respectively, and IMHO that is a mistake. We should have
*one* native wait condition resource type, one AWS-compatible one,
software deployments and any custom plugin that require signalling; and
they should all use a common SignalResponder implementation that would
call an API that is pluggable using stevedore. (In summary, what we're
trying to make configurable is an implementation that should be
invisible to the user, not an interface that is visible to the user, and
therefore the correct unit of abstraction is an API, not a resource.)
I just noticed, however, that there is an already-partially-implemented
blueprint[3] and further pending patches[4] to use stevedore for *all*
types of plugins - particularly resource plugins[5] - in Heat. I feel
very strongly that stevedore is _not_ a good fit for all of those use
cases. (Disclaimer: obviously I _would_ think that, since I implemented
the current system instead of using stevedore for precisely that reason.)
The stated benefit of switching to stevedore is that it solves issues
like https://launchpad.net/bugs/1292655 that are caused by the current
convoluted layout of /contrib. I think the layout stems at least in part
from a misunderstanding of how the current plugin_manager works. The
point of the plugin_manager is that each plugin directory does *not*
have to be a Python package - it can be any directory. Modules in the
directory then appear in the package heat.engine.plugins once imported.
So there is no need to do what we are currently doing, creating a
resources package, and then a parent package that contains the tests
package as well, and then in the tests doing:
from ..resources import docker_container ## noqa
All we really need to do is throw the resources in any old directory,
add that directory to the plugin_dirs list, stick the tests in any old
package, and from the tests do
from heat.engine.plugins import docker_container
The main reason we haven't done this seems to be to avoid having to list
the various contrib plugin dirs separately. Stevedore "solves" this by
forcing us to list not only each directory but each class in each module
in each directory separately. The tricky part of fixing the current
layout is ensuring the contrib plugin directories get added to the
plugin_dirs list during the unit tests and only during the unit tests.
However, I'm confident that could be fixed with no more difficulty than
the stevedore changes and with far less disruption to existing operators
using custom plugins.
Stevedore is ideal for configuring an implementation for a small number
of well known plug points. It does not appear to be ideal for managing
an application like Heat that comprises a vast collection of
implementations of the same interface, each bound to its own plug point.
For example, there's a subtle difference in how plugin_manager loads
external modules - by searching a list of plugin directories for Python
modules - and how stevedore does it, by loading a specified module
already in the Python path. The latter is great for selecting one of a
number of implementations that already exist in the code, but not so
great for dropping in an additional external module, which now needs to
be wrapped in a package that has to be installed in the path *and*
there's still a configuration file to edit. This is way harder for a
packager and/or operator to set up.
This approach actually precludes a number of things we know we want to
do in the future - for example it would be great if the native and AWS
resource plugins were distributed as separate subpackages so that "yum
install heat-engine" installed only the native resources, and a separate
"yum install heat-cfn-plugins" added the AWS-compatibility resources.
You can't (safely) package things that way if the installation would
involve editing a config file.
One of the main design goals of the current resource plugins was to move
the mapping from resource names to classes away from one central point
(where all of the modules were imported) and place the configuration
alongside the code it applies to. I am definitely not looking forward to
having to go look up a config file to find out what each resource is
every time I open the autoscaling module (and I do need to remind myself
_every_ time I open it), to say nothing of the constant merge conflicts
that we used to have to deal with when there was a central registry.
A central registry is also problematic for operators that modify it, who
will have a difficult, manual and potentially error-prone merge task to
perform on the config file every time they upgrade.
Constraints, I feel, are very similar to resources in this respect. I am
less concerned about template formats, since there are so few of them...
although it would be really nice to be able to install these as
subpackages too, and using stevedore appears to eliminate that as an
option :(
Intrinsic functions are a different story. I'm equally opposed to
defining them in a config file instead of near the code that implements
them, but I now think I probably made a mistake in making them pluggable
at all. (The idea to make functions pluggable pre-dated the idea of
making template formats pluggable.) The ability to plug in functions to
existing template formats is an invitation for operators to do so, and
that is a recipe for a lot of incompatible templates being released into
the world with the same version string. We should probably have each
template format return a hard-coded map of intrinsic functions, and
allow operators to create their own subclass to return a different set,
and encourage them to register said subclass with a different version
string (although we couldn't stop them from overriding the built-in
implementation if they really wanted).
Summarising, my view of the right tool for the job at each of the
various plugin interfaces:
Client plugins stevedore
Signals stevedore
Resources plugin_manager
Constraints plugin_manager
Template formats plugin_manager or maybe stevedore
Intrinsic functions neither (should be bound to template format)
At a minimum, I think this blueprint requires more discussion than it
has heretofore been subject to. I would be interested to hear what
anyone else thinks.
cheers,
Zane.
[1] https://review.openstack.org/96947
[2] https://review.openstack.org/101354
[3] https://blueprints.launchpad.net/heat/+spec/stevedore-plugins
[4]
https://review.openstack.org/#/q/status:open+project:openstack/heat+branch:master+topic:stevedore-plugins,n,z
[5] https://review.openstack.org/103044
_______________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
