Public bug reported: Hello, all.
my environment openstack version is antelope. After installed openstack platform, keystone error occur for fernet token. rotate, time sync, file info, md5sum are all the same, but keep getting errors. Apr 30 14:38:39 ict-big-os01 keystone-wsgi-public[2669300]: 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application [None req-9b6f93ff-317c-4c13-9c0b-c842a36c49c0 f28be09677bc4798a9300625ee03ac86 7ae27bad09cb4e5cb311eb094c169ae7 - - default default] Could not recognize Fernet token: keystone.exception.TokenNotFound: Could not recognize Fernet token 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application Traceback (most recent call last): 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application File "/openstack/venvs/keystone-27.1.0/lib/python3.10/site-packages/keystone/token/token_formatters.py", line 89, in unpack 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application return self.crypto.decrypt(token.encode('utf-8')) 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application File "/openstack/venvs/keystone-27.1.0/lib/python3.10/site-packages/cryptography/fernet.py", line 210, in decrypt 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application raise InvalidToken 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application cryptography.fernet.InvalidToken 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application Traceback (most recent call last): 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application File "/openstack/venvs/keystone-27.1.0/lib/python3.10/site-packages/keystone/token/providers/fernet/core.py", line 99, in validate_token 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application return self.token_formatter.validate_token(token_id) 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application File "/openstack/venvs/keystone-27.1.0/lib/python3.10/site-packages/keystone/token/token_formatters.py", line 173, in validate_token 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application serialized_payload = self.unpack(token) 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application File "/openstack/venvs/keystone-27.1.0/lib/python3.10/site-packages/keystone/token/token_formatters.py", line 91, in unpack 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application raise exception.ValidationError( 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application keystone.exception.ValidationError: Could not recognize Fernet token 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application Traceback (most recent call last): 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application File "/openstack/venvs/keystone-27.1.0/lib/python3.10/site-packages/flask/app.py", line 1820, in full_dispatch_request 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application rv = self.dispatch_request() 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application File "/openstack/venvs/keystone-27.1.0/lib/python3.10/site-packages/flask/app.py", line 1796, in dispatch_request 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application File "/openstack/venvs/keystone-27.1.0/lib/python3.10/site-packages/flask_restful/__init__.py", line 467, in wrapper 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application resp = resource(*args, **kwargs) 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application File "/openstack/venvs/keystone-27.1.0/lib/python3.10/site-packages/flask/views.py", line 107, in view 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application File "/openstack/venvs/keystone-27.1.0/lib/python3.10/site-packages/flask_restful/__init__.py", line 582, in dispatch_request 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application resp = meth(*args, **kwargs) 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application File "/openstack/venvs/keystone-27.1.0/lib/python3.10/site-packages/keystone/api/auth.py", line 285, in get 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:validate_token') 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application File "/openstack/venvs/keystone-27.1.0/lib/python3.10/site-packages/keystone/common/rbac_enforcer/enforcer.py", line 422, in enforce_call 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application subj_token_target_data = cls._extract_subject_token_target_data() 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application File "/openstack/venvs/keystone-27.1.0/lib/python3.10/site-packages/keystone/common/rbac_enforcer/enforcer.py", line 261, in _extract_subject_token_target_data 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application token = PROVIDER_APIS.token_provider_api.validate_token( 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application File "/openstack/venvs/keystone-27.1.0/lib/python3.10/site-packages/keystone/common/manager.py", line 115, in wrapped 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application File "/openstack/venvs/keystone-27.1.0/lib/python3.10/site-packages/keystone/token/provider.py", line 145, in validate_token 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application token = self._validate_token(token_id) 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application File "/openstack/venvs/keystone-27.1.0/lib/python3.10/site-packages/decorator.py", line 232, in fun 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application File "/openstack/venvs/keystone-27.1.0/lib/python3.10/site-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application return self.get_or_create( 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application File "/openstack/venvs/keystone-27.1.0/lib/python3.10/site-packages/dogpile/cache/region.py", line 1042, in get_or_create 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application with Lock( 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application File "/openstack/venvs/keystone-27.1.0/lib/python3.10/site-packages/dogpile/lock.py", line 185, in __enter__ 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application return self._enter() 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application File "/openstack/venvs/keystone-27.1.0/lib/python3.10/site-packages/dogpile/lock.py", line 94, in _enter 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application File "/openstack/venvs/keystone-27.1.0/lib/python3.10/site-packages/dogpile/lock.py", line 178, in _enter_create 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application return self.creator() 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application File "/openstack/venvs/keystone-27.1.0/lib/python3.10/site-packages/dogpile/cache/region.py", line 995, in gen_value 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application created_value = creator( 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application File "/openstack/venvs/keystone-27.1.0/lib/python3.10/site-packages/keystone/token/provider.py", line 158, in _validate_token 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application issued_at, expires_at) = self.driver.validate_token(token_id) 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application File "/openstack/venvs/keystone-27.1.0/lib/python3.10/site-packages/keystone/token/providers/fernet/core.py", line 101, in validate_token 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application raise exception.TokenNotFound(e) 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application keystone.exception.TokenNotFound: Could not recognize Fernet token 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application root@ict-big-os01:/etc/keystone/fernet-keys# ls 0 1 2 root@ict-big-os01:/etc/keystone/fernet-keys# timedatectl Local time: Wed 2025-04-30 16:05:58 KST Universal time: Wed 2025-04-30 07:05:58 UTC RTC time: Wed 2025-04-30 07:05:58 Time zone: Asia/Seoul (KST, +0900) System clock synchronized: yes NTP service: active RTC in local TZ: no root@ict-big-os01:/etc/keystone/fernet-keys# chronyc sources MS Name/IP address Stratum Poll Reach LastRx Last sample =============================================================================== ^* 10.100.88.42 4 10 377 521 -186us[ -345us] +/- 20ms ^+ 10.100.88.119 4 10 377 715 -1313us[-1471us] +/- 24ms root@ict-big-os01:/etc/keystone/fernet-keys# ls -al total 20 drwxr-s--- 2 keystone keystone 4096 Apr 30 16:02 . drwxr-xr-x 8 keystone keystone 4096 Apr 30 16:00 .. -rw------- 1 keystone keystone 44 Apr 30 16:02 0 -rw------- 1 keystone keystone 44 Apr 30 16:02 1 -rw------- 1 keystone keystone 44 Apr 30 16:02 2 root@ict-big-os01:/etc/keystone/fernet-keys# md5sum * 4c3f619e539cfc19867f57ddd714f0d7 0 875753799cc9a24c3544bd3936dd867f 1 04270ec3523eecb959a4282d83b614a1 2 root@ict-big-os02:/etc/keystone/fernet-keys# ls 0 1 2 root@ict-big-os02:/etc/keystone/fernet-keys# timedatectl Local time: Wed 2025-04-30 16:05:58 KST Universal time: Wed 2025-04-30 07:05:58 UTC RTC time: Wed 2025-04-30 07:05:58 Time zone: Asia/Seoul (KST, +0900) System clock synchronized: yes NTP service: active RTC in local TZ: no root@ict-big-os02:/etc/keystone/fernet-keys# chronyc sources MS Name/IP address Stratum Poll Reach LastRx Last sample =============================================================================== ^* 10.100.88.42 4 6 377 26 -9259ns[ -223us] +/- 20ms ^+ 10.100.88.119 4 10 377 548 -541us[ +499us] +/- 24ms root@ict-big-os02:/etc/keystone/fernet-keys# ls -al total 20 drwxr-s--- 2 keystone keystone 4096 Apr 30 16:02 . drwxr-xr-x 7 keystone keystone 4096 Apr 30 14:35 .. -rw------- 1 keystone keystone 44 Apr 30 16:02 0 -rw------- 1 keystone keystone 44 Apr 30 16:02 1 -rw------- 1 keystone keystone 44 Apr 30 16:02 2 root@ict-big-os02:/etc/keystone/fernet-keys# md5sum * 4c3f619e539cfc19867f57ddd714f0d7 0 875753799cc9a24c3544bd3936dd867f 1 04270ec3523eecb959a4282d83b614a1 2 root@ict-big-os03:/etc/keystone/fernet-keys# ls 0 1 2 root@ict-big-os03:/etc/keystone/fernet-keys# timedatectl Local time: Wed 2025-04-30 16:05:58 KST Universal time: Wed 2025-04-30 07:05:58 UTC RTC time: Wed 2025-04-30 07:05:58 Time zone: Asia/Seoul (KST, +0900) System clock synchronized: yes NTP service: active RTC in local TZ: no root@ict-big-os03:/etc/keystone/fernet-keys# chronyc sources MS Name/IP address Stratum Poll Reach LastRx Last sample =============================================================================== ^* 10.100.88.42 4 10 377 253 -544us[ -558us] +/- 21ms ^+ 10.100.88.119 4 10 377 390 -376us[ -391us] +/- 24ms root@ict-big-os03:/etc/keystone/fernet-keys# ls -al total 20 drwxr-s--- 2 keystone keystone 4096 Apr 30 16:02 . drwxr-xr-x 7 keystone keystone 4096 Apr 30 14:24 .. -rw------- 1 keystone keystone 44 Apr 30 16:02 0 -rw------- 1 keystone keystone 44 Apr 30 16:02 1 -rw------- 1 keystone keystone 44 Apr 30 16:02 2 root@ict-big-os03:/etc/keystone/fernet-keys# md5sum * 4c3f619e539cfc19867f57ddd714f0d7 0 875753799cc9a24c3544bd3936dd867f 1 04270ec3523eecb959a4282d83b614a1 2 if you hanve any questions, plz let me know. thanks. ** Affects: keystone Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Identity (keystone). https://bugs.launchpad.net/bugs/2110084 Title: [Keystone] Could not recognize Fernet token issue. Status in OpenStack Identity (keystone): New Bug description: Hello, all. my environment openstack version is antelope. After installed openstack platform, keystone error occur for fernet token. rotate, time sync, file info, md5sum are all the same, but keep getting errors. Apr 30 14:38:39 ict-big-os01 keystone-wsgi-public[2669300]: 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application [None req-9b6f93ff-317c-4c13-9c0b-c842a36c49c0 f28be09677bc4798a9300625ee03ac86 7ae27bad09cb4e5cb311eb094c169ae7 - - default default] Could not recognize Fernet token: keystone.exception.TokenNotFound: Could not recognize Fernet token 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application Traceback (most recent call last): 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application File "/openstack/venvs/keystone-27.1.0/lib/python3.10/site-packages/keystone/token/token_formatters.py", line 89, in unpack 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application return self.crypto.decrypt(token.encode('utf-8')) 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application File "/openstack/venvs/keystone-27.1.0/lib/python3.10/site-packages/cryptography/fernet.py", line 210, in decrypt 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application raise InvalidToken 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application cryptography.fernet.InvalidToken 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application Traceback (most recent call last): 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application File "/openstack/venvs/keystone-27.1.0/lib/python3.10/site-packages/keystone/token/providers/fernet/core.py", line 99, in validate_token 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application return self.token_formatter.validate_token(token_id) 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application File "/openstack/venvs/keystone-27.1.0/lib/python3.10/site-packages/keystone/token/token_formatters.py", line 173, in validate_token 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application serialized_payload = self.unpack(token) 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application File "/openstack/venvs/keystone-27.1.0/lib/python3.10/site-packages/keystone/token/token_formatters.py", line 91, in unpack 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application raise exception.ValidationError( 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application keystone.exception.ValidationError: Could not recognize Fernet token 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application Traceback (most recent call last): 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application File "/openstack/venvs/keystone-27.1.0/lib/python3.10/site-packages/flask/app.py", line 1820, in full_dispatch_request 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application rv = self.dispatch_request() 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application File "/openstack/venvs/keystone-27.1.0/lib/python3.10/site-packages/flask/app.py", line 1796, in dispatch_request 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application File "/openstack/venvs/keystone-27.1.0/lib/python3.10/site-packages/flask_restful/__init__.py", line 467, in wrapper 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application resp = resource(*args, **kwargs) 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application File "/openstack/venvs/keystone-27.1.0/lib/python3.10/site-packages/flask/views.py", line 107, in view 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application File "/openstack/venvs/keystone-27.1.0/lib/python3.10/site-packages/flask_restful/__init__.py", line 582, in dispatch_request 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application resp = meth(*args, **kwargs) 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application File "/openstack/venvs/keystone-27.1.0/lib/python3.10/site-packages/keystone/api/auth.py", line 285, in get 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:validate_token') 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application File "/openstack/venvs/keystone-27.1.0/lib/python3.10/site-packages/keystone/common/rbac_enforcer/enforcer.py", line 422, in enforce_call 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application subj_token_target_data = cls._extract_subject_token_target_data() 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application File "/openstack/venvs/keystone-27.1.0/lib/python3.10/site-packages/keystone/common/rbac_enforcer/enforcer.py", line 261, in _extract_subject_token_target_data 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application token = PROVIDER_APIS.token_provider_api.validate_token( 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application File "/openstack/venvs/keystone-27.1.0/lib/python3.10/site-packages/keystone/common/manager.py", line 115, in wrapped 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application File "/openstack/venvs/keystone-27.1.0/lib/python3.10/site-packages/keystone/token/provider.py", line 145, in validate_token 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application token = self._validate_token(token_id) 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application File "/openstack/venvs/keystone-27.1.0/lib/python3.10/site-packages/decorator.py", line 232, in fun 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application File "/openstack/venvs/keystone-27.1.0/lib/python3.10/site-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application return self.get_or_create( 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application File "/openstack/venvs/keystone-27.1.0/lib/python3.10/site-packages/dogpile/cache/region.py", line 1042, in get_or_create 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application with Lock( 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application File "/openstack/venvs/keystone-27.1.0/lib/python3.10/site-packages/dogpile/lock.py", line 185, in __enter__ 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application return self._enter() 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application File "/openstack/venvs/keystone-27.1.0/lib/python3.10/site-packages/dogpile/lock.py", line 94, in _enter 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application File "/openstack/venvs/keystone-27.1.0/lib/python3.10/site-packages/dogpile/lock.py", line 178, in _enter_create 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application return self.creator() 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application File "/openstack/venvs/keystone-27.1.0/lib/python3.10/site-packages/dogpile/cache/region.py", line 995, in gen_value 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application created_value = creator( 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application File "/openstack/venvs/keystone-27.1.0/lib/python3.10/site-packages/keystone/token/provider.py", line 158, in _validate_token 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application issued_at, expires_at) = self.driver.validate_token(token_id) 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application File "/openstack/venvs/keystone-27.1.0/lib/python3.10/site-packages/keystone/token/providers/fernet/core.py", line 101, in validate_token 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application raise exception.TokenNotFound(e) 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application keystone.exception.TokenNotFound: Could not recognize Fernet token 2025-04-30 14:38:39.804 2669300 ERROR keystone.server.flask.application root@ict-big-os01:/etc/keystone/fernet-keys# ls 0 1 2 root@ict-big-os01:/etc/keystone/fernet-keys# timedatectl Local time: Wed 2025-04-30 16:05:58 KST Universal time: Wed 2025-04-30 07:05:58 UTC RTC time: Wed 2025-04-30 07:05:58 Time zone: Asia/Seoul (KST, +0900) System clock synchronized: yes NTP service: active RTC in local TZ: no root@ict-big-os01:/etc/keystone/fernet-keys# chronyc sources MS Name/IP address Stratum Poll Reach LastRx Last sample =============================================================================== ^* 10.100.88.42 4 10 377 521 -186us[ -345us] +/- 20ms ^+ 10.100.88.119 4 10 377 715 -1313us[-1471us] +/- 24ms root@ict-big-os01:/etc/keystone/fernet-keys# ls -al total 20 drwxr-s--- 2 keystone keystone 4096 Apr 30 16:02 . drwxr-xr-x 8 keystone keystone 4096 Apr 30 16:00 .. -rw------- 1 keystone keystone 44 Apr 30 16:02 0 -rw------- 1 keystone keystone 44 Apr 30 16:02 1 -rw------- 1 keystone keystone 44 Apr 30 16:02 2 root@ict-big-os01:/etc/keystone/fernet-keys# md5sum * 4c3f619e539cfc19867f57ddd714f0d7 0 875753799cc9a24c3544bd3936dd867f 1 04270ec3523eecb959a4282d83b614a1 2 root@ict-big-os02:/etc/keystone/fernet-keys# ls 0 1 2 root@ict-big-os02:/etc/keystone/fernet-keys# timedatectl Local time: Wed 2025-04-30 16:05:58 KST Universal time: Wed 2025-04-30 07:05:58 UTC RTC time: Wed 2025-04-30 07:05:58 Time zone: Asia/Seoul (KST, +0900) System clock synchronized: yes NTP service: active RTC in local TZ: no root@ict-big-os02:/etc/keystone/fernet-keys# chronyc sources MS Name/IP address Stratum Poll Reach LastRx Last sample =============================================================================== ^* 10.100.88.42 4 6 377 26 -9259ns[ -223us] +/- 20ms ^+ 10.100.88.119 4 10 377 548 -541us[ +499us] +/- 24ms root@ict-big-os02:/etc/keystone/fernet-keys# ls -al total 20 drwxr-s--- 2 keystone keystone 4096 Apr 30 16:02 . drwxr-xr-x 7 keystone keystone 4096 Apr 30 14:35 .. -rw------- 1 keystone keystone 44 Apr 30 16:02 0 -rw------- 1 keystone keystone 44 Apr 30 16:02 1 -rw------- 1 keystone keystone 44 Apr 30 16:02 2 root@ict-big-os02:/etc/keystone/fernet-keys# md5sum * 4c3f619e539cfc19867f57ddd714f0d7 0 875753799cc9a24c3544bd3936dd867f 1 04270ec3523eecb959a4282d83b614a1 2 root@ict-big-os03:/etc/keystone/fernet-keys# ls 0 1 2 root@ict-big-os03:/etc/keystone/fernet-keys# timedatectl Local time: Wed 2025-04-30 16:05:58 KST Universal time: Wed 2025-04-30 07:05:58 UTC RTC time: Wed 2025-04-30 07:05:58 Time zone: Asia/Seoul (KST, +0900) System clock synchronized: yes NTP service: active RTC in local TZ: no root@ict-big-os03:/etc/keystone/fernet-keys# chronyc sources MS Name/IP address Stratum Poll Reach LastRx Last sample =============================================================================== ^* 10.100.88.42 4 10 377 253 -544us[ -558us] +/- 21ms ^+ 10.100.88.119 4 10 377 390 -376us[ -391us] +/- 24ms root@ict-big-os03:/etc/keystone/fernet-keys# ls -al total 20 drwxr-s--- 2 keystone keystone 4096 Apr 30 16:02 . drwxr-xr-x 7 keystone keystone 4096 Apr 30 14:24 .. -rw------- 1 keystone keystone 44 Apr 30 16:02 0 -rw------- 1 keystone keystone 44 Apr 30 16:02 1 -rw------- 1 keystone keystone 44 Apr 30 16:02 2 root@ict-big-os03:/etc/keystone/fernet-keys# md5sum * 4c3f619e539cfc19867f57ddd714f0d7 0 875753799cc9a24c3544bd3936dd867f 1 04270ec3523eecb959a4282d83b614a1 2 if you hanve any questions, plz let me know. thanks. To manage notifications about this bug go to: https://bugs.launchpad.net/keystone/+bug/2110084/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp