On 27/10/14 10:09, Ken Thomas wrote:
I opened https://bugs.launchpad.net/heat/+bug/1386213 to track this issue.
I understand that our situation may be a bit unusual and that changing
the schema may not be the optimum solution as it affects everyone. (As I
mentioned in the bug, we're doing an internal cloud and we're required
to use an internal user validation system that returns cookies; that's
why our 'passwords' are much bigger than y'all probably expected.)
At a minimum, we need to handle this error better, so there is
definitely something there for us to fix. Many thanks for tracking this
down, it is much appreciated!
Any suggestions for our specific situation would be greatly appreciated.
I'm thinking our best bet at this point is to have a local patch to
alter the user_creds.password column.
Your very _best_ bet is to configure Heat to use Keystone trusts and
therefore not need to store the user's password at all. Steve Hardy has
an excellent blog post on how to do it:
http://hardysteven.blogspot.co.uk/2014/04/heat-auth-model-updates-part-1-trusts.html
If that's not possible for whatever reason then increasing the column
size in a local patch is probably a decent solution, although it has a
small potential to cause problems with database migrations in the future.
cheers,
Zane.
Ken
------------------------------------------------------------------------
*From:* Ken Thomas <k...@yahoo-inc.com>
*To:* Zane Bitter <zbit...@redhat.com>; "openstack@lists.openstack.org"
<openstack@lists.openstack.org>
*Sent:* Friday, October 24, 2014 3:33 PM
*Subject:* Re: [Openstack] [openstack-dev] [Heat] Stack stuck in
DELETE_IN_PROGRESS even though all resources are DELETE_COMPLETE
Found it! The user_creds.password column in the database is 255
characters. My value is long enough (800+) that it fills it up and gets
truncated. I think that garbling of the data leads to the padding
exception.
If I modify the database so that the column is longer, then the delete
goes through just fine.
Ken
------------------------------------------------------------------------
*From:* Ken Thomas <k...@yahoo-inc.com>
*To:* Zane Bitter <zbit...@redhat.com>; "openstack@lists.openstack.org"
<openstack@lists.openstack.org>
*Sent:* Friday, October 24, 2014 2:43 PM
*Subject:* Re: [openstack-dev] [Heat] Stack stuck in DELETE_IN_PROGRESS
even though all resources are DELETE_COMPLETE
I've got some additional info for y'all on this. The problem is in
heat.engine.parser.Stack.delete with this statement:
user_creds = db_api.user_creds_get(self.user_creds_id)
In my case, that call is throwing an exception that's not getting caught
or logged. The exact exception is:
ERROR Incorrect padding
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/heat/engine/parser.py", line
670, in delete
user_creds = db_api.user_creds_get(self.user_creds_id)
File "/usr/lib/python2.6/site-packages/heat/db/api.py", line 170, in
user_creds_get
return IMPL.user_creds_get(context_id)
File "/usr/lib/python2.6/site-packages/heat/db/sqlalchemy/api.py",
line 468, in user_creds_get
result['password'] = _decrypt(result['password'],
db_result.decrypt_method)
File "/usr/lib/python2.6/site-packages/heat/db/sqlalchemy/api.py",
line 174, in _decrypt
value = decryptor(enc_value)
File "/usr/lib/python2.6/site-packages/heat/common/crypt.py", line
48, in oslo_decrypt_v1
auth_info, b64decode=True)
File
"/usr/lib/python2.6/site-packages/heat/openstack/common/crypto/utils.py", line
158, in decrypt
msg = base64.b64decode(msg)
File "/usr/lib64/python2.6/base64.py", line 76, in b64decode
raise TypeError(msg)
TypeError: Incorrect padding
Does this ring any bells? I checked the user_creds table and it looks
okay as far as I can tell. Do I have a mismatch between how the value is
encoded vs how it's being decoded?
Ken
------------------------------------------------------------------------
*From:* Zane Bitter <zbit...@redhat.com>
*To:* Ken Thomas <k...@yahoo-inc.com>; "openstack@lists.openstack.org"
<openstack@lists.openstack.org>
*Sent:* Wednesday, October 22, 2014 6:10 AM
*Subject:* Re: [openstack-dev] [Heat] Stack stuck in DELETE_IN_PROGRESS
even though all resources are DELETE_COMPLETE
Moving this to the openstack@ list...
On 21/10/14 18:56, Ken Thomas wrote:
> Greetings all,
> I'm using Heat from Icehouse and I'm hitting a problem that I'm
hoping y'all can shed some light on.
> I have no problems doing stack-create. I can watch the MySQL commands
go by and see it happily update the stack table so that it eventually
shows up as CREATE_COMPLETE. When I delete the stack, everything seems
to be working fine. I see the MySQL update that sets the stack to
DELETE IN PROGRESS as well as the updates that sets my single resource
to DELETE_COMPLETE... but I never see the final update to the stack
table to set it to DELETE COMPLETE.
I don't think you're the first person to report this, but afaik we've
never been able to track it down. There are a bunch of things that Heat
does after deleting the resources, like cleaning up trusts in Keystone,
and perhaps one of those is failing. However, *in theory* there's no way
for the stack to be left in the "DELETE_IN_PROGRESS" state without an
exception being raised that would leave a prominent entry in the logs.
> One very odd thing that I found in the MySQL logs (snippet below), is
a query that includes that stack name but with an extra '*' character
append to it. My stack is named 'xyzzy8' but notice the 'xzyzzy8*' in
the constraint. That's not going to return anything and I'm wondering if
that is what's preventing the final stack DELETE_COMPLTE update from
happening?
The stack name that ends with '*' is the "backup stack", which is used
to hold out-of-date resources during a stack update. When the stack is
deleted, Heat will check for the existence of the backup stack and make
sure any resources left behind in there (from a failed update) are
deleted too. If it doesn't exist, that shouldn't be a problem.
> There are no errors in the any of the heat logs.
>
> Any guidance would be greatly appreciated!
All I can suggest is to (a) make sure you have the very latest version
of Icehouse and (b) make sure your logging is turned up to 11, then try
again.
Since you seem very proficient at debugging, if that doesn't work you
could also try adding extra logging into the method
heat.engine.parser.Stack.delete to try to figure out how far it is getting.
Feel free to raise a bug in Launchpad and attach any logs that might
help us figure out what is happening.
cheers,
Zane.
>
> Ken
> SELECT stack.status_reason AS stack_status_reason, stack.created_at
AS stack_created_at, stack.deleted_at AS stack_deleted_at, stack.action
AS stack_action, stack.status AS stack_status, stack.id AS stack_id,
stack.name AS stack_name, stack.raw_template_id AS
stack_raw_template_id, stack.username AS stack_username, stack.tenant AS
stack_tenant, stack.parameters AS stack_parameters, stack.user_creds_id
AS stack_user_creds_id, stack.owner_id AS stack_owner_id, stack.timeout
AS stack_timeout, stack.disable_rollback AS stack_disable_rollback,
stack.stack_user_project_id AS stack_stack_user_project_id,
stack.updated_at AS stack_updated_atFROM stackWHERE stack.deleted_at IS
NULL AND (stack.tenant = 'c6c488223aae4e97bf56dda8cef36b3b' OR
stack.stack_user_project_id = 'c6c488223aae4e97bf56dda8cef36b3b') AND
stack.name = 'xyzzy8*' AND stack.owner_id =
'9a3e56d7-0c10-4c1c-8c54-0e5580cee121'
_______________________________________________
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
<mailto:openstack@lists.openstack.org>
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
_______________________________________________
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
