On 16/08/13 00:50, Christopher Armstrong wrote:
*Introduction and Requirements*
So there's kind of a perfect storm happening around autoscaling in Heat
right now. It's making it really hard to figure out how I should compose
this email. There are a lot of different requirements, a lot of
different cool ideas, and a lot of projects that want to take advantage
of autoscaling in one way or another: Trove, OpenShift, TripleO, just to
name a few...
I'll try to list the requirements from various people/projects that may
be relevant to autoscaling or scaling in general.
1. Some users want a service like Amazon's Auto Scaling or Rackspace's
Otter -- a simple API that doesn't really involve orchestration.
2. If such a API exists, it makes sense for Heat to take advantage of
its functionality instead of reimplementing it.
+1, obviously. But the other half of the story is that the API is likely
be implemented using Heat on the back end, amongst other reasons because
that implementation already exists. (As you know, since you wrote it ;)
So, just as we will have an RDS resource in Heat that calls Trove, and
Trove will use Heat for orchestration:
user => [Heat =>] Trove => Heat => Nova
there will be a similar workflow for Autoscaling:
user => [Heat =>] Autoscaling -> Heat => Nova
where the first, optional, Heat stack contains the RDS/Autoscaling
resource and the backend Heat stack contains the actual Nova instance(s).
One difference might be that the Autoscaling -> Heat step need not
happen via the public ReST API. Since both are part of the Heat project,
I think it would also be OK to do this over RPC only.
3. If Heat integrates with that separate API, however, that API will
need two ways to do its work:
Wut?
1. native instance-launching functionality, for the "simple" use
This is just the simplest possible case of 3.2. Why would we maintain a
completely different implementation?
2. a way to talk back to Heat to perform orchestration-aware scaling
operations.
[IRC discussions clarified this to mean scaling arbitrary resource
types, rather than just Nova servers.]
4. There may be things that are different than AWS::EC2::Instance that
we would want to scale (I have personally been playing around with the
concept of a ResourceGroup, which would maintain a nested stack of
resources based on an arbitrary template snippet).
5. Some people would like to be able to perform manual operations on an
instance group -- such as Clint Byrum's recent example of "remove
instance 4 from resource group A".
Please chime in with your additional requirements if you have any! Trove
and TripleO people, I'm looking at you :-)
*TL;DR*
Point 3.2. above is the main point of this email: exactly how should the
autoscaling API talk back to Heat to tell it to add more instances? I
included the other points so that we keep them in mind while considering
a solution.
*Possible Solutions*
I have heard at least three possibilities so far:
1. the autoscaling API should maintain a full template of all the nodes
in the autoscaled nested stack, manipulate it locally when it wants to
add or remove instances, and post an update-stack to the nested-stack
associated with the InstanceGroup.
This is what I had been thinking.
Pros: It doesn't require any changes to Heat.
Cons: It puts a lot of burden of state management on the autoscale API,
All other APIs need to manage state too, I don't really have a problem
with that. It already has to handle e.g. the cooldown state; your
scaling strategy (uh, for the service) will be determined by that.
and it arguably spreads out the responsibility of "orchestration" to the
autoscale API.
Another line of argument would be that this is not true by definition ;)
Also arguable is that automated agents outside of Heat
shouldn't be managing an "internal" template, which are typically
developed by devops people and kept in version control.
2. There should be a new custom-built API for doing exactly what the
autoscaling service needs on an InstanceGroup, named something
unashamedly specific -- like "instance-group-adjust".
+1 to having a custom (RPC-only) API if it means forcing some state out
of the autoscaling service.
-1 for it talking to an InstanceGroup - that just brings back all our
old problems about having "resources" that don't have their own separate
state and APIs, but just exist inside of Heat plugins. Those are the
cause of all of the biggest design problems in Heat. They're the thing I
want the Autoscaling API to get rid of. (Also, see below.)
Pros: It'll do exactly what it needs to do for this use case; very
little state management in autoscale API; it lets Heat do all the
orchestration and only give very specific delegation to the external
autoscale API.
Cons: The API grows an additional method for a specific use case.
3. the autoscaling API should update the "Size" Property of the
InstanceGroup resource in the stack that it is placed in. This would
require the ability to PATCH a specific piece of a template (an
operation isomorphic to update-stack).
-1
Half the point of a separate autoscaling API is that an autoscaling
group shouldn't _need_ to exist only in the context of a template.
So, to get around this I think you're proposing something where the two
ways of interacting with Autoscaling are (with and without Heat):
user => Heat <=> Autoscaling
\
------------------> Heat => Nova
user => Autoscaling => Heat -> Heat => Nova
Instead of:
user => Heat => Autoscaling => Heat => Nova
user =========> Autoscaling => Heat => Nova
This is two lots of code to write & test and, from the user's point of
view, a bizarre and unprecedented conceptual inversion of where
Orchestration belongs in the system. (Note that the backend Heat is an
implementation detail hidden from the user in either case.)
cheers,
Zane.
Pros: The API modification is generic, simply a more optimized version
of update-stack; very little state management required in autoscale API.
Cons: This would essentially require manipulating the user-provided
template. (unless we have a concept of "private properties", which
perhaps wouldn't appear in the template as provided by the user, but can
be manipulated with such an update stack operation?)
*Addenda*
Keep in mind that there are use cases which require other types of
manipulation of the InstanceGroup -- not just the autoscaling API. For
example, see Clint's #5 above.
Also, about implementation: Andrew Plunk and I have begun work on Heat
resources for Rackspace's Otter, which I think will be a really good
proof of concept for how this stuff should work in the Heat-native
autoscale API. I am trying to gradually work the design into the native
Heat autoscaling design, and we will need to solve the
autoscale-controlling-InstanceGroup issue soon.
--
IRC: radix
Christopher Armstrong
Rackspace
_______________________________________________
OpenStack-dev mailing list
[email protected]
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
_______________________________________________
OpenStack-dev mailing list
[email protected]
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
