We discussed $SUBJECT at the summit as one of the main performance
problems that people are running into when trying to create very large
autoscaling groups, as projects like Sahara, Magnum, TripleO, OpenShift
are wont to do. Of course, as we all know, validation happens
synchronously, so it's prone to causing RPC timeouts that mean a hard
failure of the parent stack.
First the good news - I just committed this patch:
https://review.openstack.org/#/c/400961/
which should mean from now on that resources with identical definitions
will not all be validated, and instead we'll just validate one
representative one. In theory this should mean that autoscaling groups
should now validate in constant rather than linear time. If anyone from
one of the affected projects is able to confirm this, then I'd be happy
to backport the patch to stable/newton. It really is very simple.
The bad news here is for users of ResourceGroups with %index%
substitution (*cough*TripleO*cough*) - this makes each resource
definition unique, so it won't benefit from this fix. (Adding this to my
mental list of reasons why index substitution is bad.)
I also investigated another issue, which is that since the fix for
https://bugs.launchpad.net/heat/+bug/1388140 landed (in Kilo) I believe
we are validating nested stacks multiple times (specifically, m times,
where m is the stack's depth in the tree):
root child grandchild
create
-> validate ----------> validate --------------> validate
-> Resource.create ===> create
-> validate ----------> validate
-> Resource.create ===> create
-> validate
The only good news here is that ResourceGroup is smart enough to make
sure that it generates a nested stack with at most 1 resource to
validate when validate() is called. (However, when the nested stack is
created, and thus validated, it is of course full-sized.) Autoscaling
groups make no such allowances, but the patch above should actually have
the same effect. (We can't get rid of the special case for ResourceGroup
though, because of index substitution.)
An obvious fix would be to disable validation - or, more specifically,
validation of _resources_ - on create/update for stacks that have a
non-null owner_id (i.e. nested stacks), so that we had something like:
root child grandchild
create
-> validate ----------> validate --------------> validate
-> Resource.create ===> create
-> Resource.create ===> create
That would eliminate the duplication/triplication/multiplication of
validation. It would also mean that we'd cut out the expensive part of
ResourceGroup validation with index substitution, leaving only the cheap
part.
One downside is that in the ResourceGroup/index substitution case we'd
be creating resources whose definitions hadn't _ever_ been validated. I
_think_ that's safe, in the sense that you'd just hear about errors
later, as opposed to everything falling over in a heap, but it's
difficult to be certain. Hearing about problems late is also not ideal
(since it may cause otherwise-healthy siblings to be cancelled), but I
would guess that heavy users like TripleO developers would say that it's
worth the tradeoff.
However, one other thing about this bothers me. The part of validation
that we're keeping:
-> validate ----------> validate --------------> validate
involves loading all of the nested stacks in memory at once (i.e. the
thing we were not supposed to be doing any more in Kilo, in favour of
farming nested stacks out over RPC.) As we discovered when we found out
we were doing the same thing with outputs[1], this is a bit like hanging
out a giant "Kick Me" sign for the OOM Killer.
That's mitigated quite a lot by my patch though... we'll load the whole
autoscaling group stack in memory, but if its members are themselves
nested stacks we'll load only one of them. So the scaling tendencies
will hopefully be dominated by the complexity of your templates more
than than the size of your deployment. ResourceGroup is in a better
position, because its nested stack will actually have only one member,
so the size shouldn't affect memory consumption at all during validation.
Some options:
1) Chalk it up to an acceptable tradeoff
2) Add a single-member special case for autoscaling group validation
3) Farm out the nested validation over RPC
4) Both (2) & (3)
5) Some totally different arrangement of how nested stacks are validated
Discuss.
cheers,
Zane.
[1] https://review.openstack.org/#/c/383839/
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
