** Also affects: keystone
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Identity (keystone).
https://bugs.launchpad.net/bugs/1832005
Title:
Race during Keystone deploy (fernet)
Status in OpenStack Identity (keystone):
New
Status in kolla-ansible:
New
Bug description:
RedHat 7.6 OpenStack Ocata
Custom build Docker images using binary type.
Keystone configured to use fernet tokens.
When keystone container is started it expects directory and tokens to be
present.
This is checked by the following code
https://github.com/openstack/keystone/blob/3d2b293d7edfb0bd4bdec9b33abc63d1308e10bd/keystone/token/providers/fernet/core.py#L36
In some rare scenarios, keystone container fails with
2019-05-31 17:26:39.620011 File
"/usr/lib/python2.7/site-packages/keystone/token/providers/fernet/core.py",
line 45, in _init_
2019-05-31 17:26:39.620106 'Fernet keys.') % subs)
2019-05-31 17:26:39.620126 SystemExit: /etc/keystone/fernet-keys/ does not
contain keys, use keystone-manage fernet_setup to create Fernet keys.
When inspecting directory, keys are there
(keystone)[root@osc1 fernet-keys]# ls -la
total 12
drwxrwx---. 2 keystone keystone 33 May 31 17:26 .
drwxr-x---. 1 root keystone 61 May 31 17:26 ..
rw------. 1 keystone keystone 44 May 31 17:26 0
rw------. 1 keystone keystone 44 May 31 17:26 1
rw------. 1 keystone keystone 44 May 31 17:26 2
Please note that the files creation time is the same as error message
time (17:26).
Upon inspection of the ansible/roles/keystone/tasks/deploy.yml one can find
that
init_fernet.yml task is executed after flush_handlers. When handlers are run,
containers are created or restarted.
The obvious option would be to move init_fernet before handlers, but
this task does require keystone_ssh and keystone_fernet to be up and
running.
The solutions could include:
- Changes in keystone itself to retry initialization as long as the keys are
missing
- Changes in keystone to fail in a way that the container will restart
- Changes in kolla-ansible to enforce fernet init before keystone container
starts.
The bug is found on Ocata but upon Ansible manifests inspection it could
happen on master as well.
Workaround:
Restart Keystone container.
To manage notifications about this bug go to:
https://bugs.launchpad.net/keystone/+bug/1832005/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
