Thanks for the background info! I'll see what I can suss out since it's easily
(100%) reproducable for me.
I'll keep y'all posted although it may be a day or two before I can get back to
it.
Ken
From: Zane Bitter <zbit...@redhat.com>
To: Ken Thomas <k...@yahoo-inc.com>; "openstack@lists.openstack.org"
<openstack@lists.openstack.org>
Sent: Wednesday, October 22, 2014 6:10 AM
Subject: Re: [openstack-dev] [Heat] Stack stuck in DELETE_IN_PROGRESS even
though all resources are DELETE_COMPLETE
Moving this to the openstack@ list...
On 21/10/14 18:56, Ken Thomas wrote:
> Greetings all,
> I'm using Heat from Icehouse and I'm hitting a problem that I'm hoping y'all
> can shed some light on.
> I have no problems doing stack-create. I can watch the MySQL commands go by
> and see it happily update the stack table so that it eventually shows up as
> CREATE_COMPLETE. When I delete the stack, everything seems to be working
> fine. I see the MySQL update that sets the stack to DELETE IN PROGRESS as
> well as the updates that sets my single resource to DELETE_COMPLETE... but I
> never see the final update to the stack table to set it to DELETE COMPLETE.
I don't think you're the first person to report this, but afaik we've
never been able to track it down. There are a bunch of things that Heat
does after deleting the resources, like cleaning up trusts in Keystone,
and perhaps one of those is failing. However, *in theory* there's no way
for the stack to be left in the "DELETE_IN_PROGRESS" state without an
exception being raised that would leave a prominent entry in the logs.
> One very odd thing that I found in the MySQL logs (snippet below), is a query
> that includes that stack name but with an extra '*' character append to it.
> My stack is named 'xyzzy8' but notice the 'xzyzzy8*' in the constraint.
> That's not going to return anything and I'm wondering if that is what's
> preventing the final stack DELETE_COMPLTE update from happening?
The stack name that ends with '*' is the "backup stack", which is used
to hold out-of-date resources during a stack update. When the stack is
deleted, Heat will check for the existence of the backup stack and make
sure any resources left behind in there (from a failed update) are
deleted too. If it doesn't exist, that shouldn't be a problem.
> There are no errors in the any of the heat logs.
>
> Any guidance would be greatly appreciated!
All I can suggest is to (a) make sure you have the very latest version
of Icehouse and (b) make sure your logging is turned up to 11, then try
again.
Since you seem very proficient at debugging, if that doesn't work you
could also try adding extra logging into the method
heat.engine.parser.Stack.delete to try to figure out how far it is getting.
Feel free to raise a bug in Launchpad and attach any logs that might
help us figure out what is happening.
cheers,
Zane.
>
> Ken
> SELECT stack.status_reason AS stack_status_reason, stack.created_at AS
> stack_created_at, stack.deleted_at AS stack_deleted_at, stack.action AS
> stack_action, stack.status AS stack_status, stack.id AS stack_id, stack.name
> AS stack_name, stack.raw_template_id AS stack_raw_template_id, stack.username
> AS stack_username, stack.tenant AS stack_tenant, stack.parameters AS
> stack_parameters, stack.user_creds_id AS stack_user_creds_id, stack.owner_id
> AS stack_owner_id, stack.timeout AS stack_timeout, stack.disable_rollback AS
> stack_disable_rollback, stack.stack_user_project_id AS
> stack_stack_user_project_id, stack.updated_at AS stack_updated_atFROM
> stackWHERE stack.deleted_at IS NULL AND (stack.tenant =
> 'c6c488223aae4e97bf56dda8cef36b3b' OR stack.stack_user_project_id =
> 'c6c488223aae4e97bf56dda8cef36b3b') AND stack.name = 'xyzzy8*' AND
> stack.owner_id = '9a3e56d7-0c10-4c1c-8c54-0e5580cee121'
_______________________________________________
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
