<< Testing keystone revert in https://review.opendev.org/c/openstack/neutron/+/925968 The job passed with the revert.
<< Looks like the issue is with osprofiler being enabled for keystone in the neutron gate, but it's not being enabled in keystone gates, so that's why we didn't see the issue there. @xek yes exactly that's why it got missed. Wouldn't be good to have some coverage in keystone gate as well for this? << So the fix should be to extend oslo_serialization to support date type and not to revert keystone patch. Ok i can push a patch in oslo.serialization, but i also noticed the keystone patch is also being backported so oslo.serialization fixes would also require backports and releases and u-c update and that can take time so if this is the way would be good to hold keystone backports. << It is actually interesting that lots of other functional tests are not failing, that means the job is running some specific setup including osprofiler @gtema it's not doing anything extra than enabling osprofiler, for test i did enabled it in devstack jobs and all those too failed[1][2] with same issue. [1] https://review.opendev.org/c/openstack/devstack/+/926169/1 [2] https://zuul.opendev.org/t/openstack/buildset/b80b78109f344777afe00d02d62ef5fa ** Also affects: oslo.serialization Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/2076430 Title: rally job broken with recent keystone change and fails with ValueError: Cannot convert datetime.date(2024, 8, 7) to primitive Status in neutron: New Status in oslo.serialization: In Progress Bug description: Test fails as:- 2024-08-07 17:06:05.650302 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients [-] Unable to authenticate for user c_rally_927546a8_h6aDLbnK in project c_rally_927546a8_ahUUmlp9: keystoneauth1.exceptions.http.InternalServerError: Internal Server Error (HTTP 500) 2024-08-07 17:06:05.650941 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients Traceback (most recent call last): 2024-08-07 17:06:05.651113 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients File "/opt/stack/rally-openstack/rally_openstack/common/osclients.py", line 269, in auth_ref 2024-08-07 17:06:05.651156 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients self.cache["keystone_auth_ref"] = plugin.get_access(sess) 2024-08-07 17:06:05.651193 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients File "/opt/stack/data/venv/lib/python3.10/site-packages/keystoneauth1/identity/base.py", line 131, in get_access 2024-08-07 17:06:05.651229 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients self.auth_ref = self.get_auth_ref(session) 2024-08-07 17:06:05.651263 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients File "/opt/stack/data/venv/lib/python3.10/site-packages/keystoneauth1/identity/generic/base.py", line 205, in get_auth_ref 2024-08-07 17:06:05.651334 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients return self._plugin.get_auth_ref(session, **kwargs) 2024-08-07 17:06:05.651348 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients File "/opt/stack/data/venv/lib/python3.10/site-packages/keystoneauth1/identity/v3/base.py", line 185, in get_auth_ref 2024-08-07 17:06:05.651356 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients resp = session.post(token_url, json=body, headers=headers, 2024-08-07 17:06:05.651363 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients File "/opt/stack/data/venv/lib/python3.10/site-packages/keystoneauth1/session.py", line 1162, in post 2024-08-07 17:06:05.651370 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients return self.request(url, 'POST', **kwargs) 2024-08-07 17:06:05.651377 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients File "/opt/stack/data/venv/lib/python3.10/site-packages/keystoneauth1/session.py", line 985, in request 2024-08-07 17:06:05.651384 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients raise exceptions.from_response(resp, method, url) 2024-08-07 17:06:05.651391 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients keystoneauth1.exceptions.http.InternalServerError: Internal Server Error (HTTP 500) From keyston log:- Aug 07 17:03:06.408072 np0038147114 [email protected][56399]: CRITICAL keystone [None req-96647e8e-2585-4279-80fa-c4fa97b8c455 None None] Unhandled error: ValueError: Cannot convert datetime.date(2024, 8, 7) to primitive Aug 07 17:03:06.408072 np0038147114 [email protected][56399]: ERROR keystone Traceback (most recent call last): Aug 07 17:03:06.408072 np0038147114 [email protected][56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask/app.py", line 1498, in __call__ Aug 07 17:03:06.408072 np0038147114 [email protected][56399]: ERROR keystone return self.wsgi_app(environ, start_response) Aug 07 17:03:06.408072 np0038147114 [email protected][56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/werkzeug/middleware/proxy_fix.py", line 183, in __call__ Aug 07 17:03:06.408072 np0038147114 [email protected][56399]: ERROR keystone return self.app(environ, start_response) Aug 07 17:03:06.408072 np0038147114 [email protected][56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py", line 129, in __call__ Aug 07 17:03:06.408072 np0038147114 [email protected][56399]: ERROR keystone resp = self.call_func(req, *args, **kw) Aug 07 17:03:06.408072 np0038147114 [email protected][56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py", line 193, in call_func Aug 07 17:03:06.408072 np0038147114 [email protected][56399]: ERROR keystone return self.func(req, *args, **kwargs) Aug 07 17:03:06.408072 np0038147114 [email protected][56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_middleware/base.py", line 124, in __call__ Aug 07 17:03:06.408072 np0038147114 [email protected][56399]: ERROR keystone response = req.get_response(self.application) Aug 07 17:03:06.408072 np0038147114 [email protected][56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/request.py", line 1313, in send Aug 07 17:03:06.408072 np0038147114 [email protected][56399]: ERROR keystone status, headers, app_iter = self.call_application( Aug 07 17:03:06.408072 np0038147114 [email protected][56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/request.py", line 1278, in call_application Aug 07 17:03:06.408072 np0038147114 [email protected][56399]: ERROR keystone app_iter = application(self.environ, start_response) Aug 07 17:03:06.408072 np0038147114 [email protected][56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py", line 143, in __call__ Aug 07 17:03:06.408072 np0038147114 [email protected][56399]: ERROR keystone return resp(environ, start_response) Aug 07 17:03:06.408072 np0038147114 [email protected][56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py", line 129, in __call__ Aug 07 17:03:06.409468 np0038147114 [email protected][56399]: ERROR keystone resp = self.call_func(req, *args, **kw) Aug 07 17:03:06.409468 np0038147114 [email protected][56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py", line 193, in call_func Aug 07 17:03:06.409468 np0038147114 [email protected][56399]: ERROR keystone return self.func(req, *args, **kwargs) Aug 07 17:03:06.409468 np0038147114 [email protected][56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_middleware/base.py", line 124, in __call__ Aug 07 17:03:06.409468 np0038147114 [email protected][56399]: ERROR keystone response = req.get_response(self.application) Aug 07 17:03:06.409468 np0038147114 [email protected][56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/request.py", line 1313, in send Aug 07 17:03:06.409468 np0038147114 [email protected][56399]: ERROR keystone status, headers, app_iter = self.call_application( Aug 07 17:03:06.409468 np0038147114 [email protected][56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/request.py", line 1278, in call_application Aug 07 17:03:06.409468 np0038147114 [email protected][56399]: ERROR keystone app_iter = application(self.environ, start_response) Aug 07 17:03:06.409468 np0038147114 [email protected][56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py", line 129, in __call__ Aug 07 17:03:06.409468 np0038147114 [email protected][56399]: ERROR keystone resp = self.call_func(req, *args, **kw) Aug 07 17:03:06.409468 np0038147114 [email protected][56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py", line 193, in call_func Aug 07 17:03:06.409468 np0038147114 [email protected][56399]: ERROR keystone return self.func(req, *args, **kwargs) Aug 07 17:03:06.409468 np0038147114 [email protected][56399]: ERROR keystone File "/opt/stack/osprofiler/osprofiler/web.py", line 131, in __call__ Aug 07 17:03:06.409468 np0038147114 [email protected][56399]: ERROR keystone return request.get_response(self.application) Aug 07 17:03:06.409468 np0038147114 [email protected][56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/request.py", line 1313, in send Aug 07 17:03:06.409468 np0038147114 [email protected][56399]: ERROR keystone status, headers, app_iter = self.call_application( Aug 07 17:03:06.409468 np0038147114 [email protected][56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/request.py", line 1278, in call_application Aug 07 17:03:06.409468 np0038147114 [email protected][56399]: ERROR keystone app_iter = application(self.environ, start_response) Aug 07 17:03:06.409468 np0038147114 [email protected][56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py", line 129, in __call__ Aug 07 17:03:06.410473 np0038147114 [email protected][56399]: ERROR keystone resp = self.call_func(req, *args, **kw) Aug 07 17:03:06.410473 np0038147114 [email protected][56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py", line 193, in call_func Aug 07 17:03:06.410473 np0038147114 [email protected][56399]: ERROR keystone return self.func(req, *args, **kwargs) Aug 07 17:03:06.410473 np0038147114 [email protected][56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_middleware/request_id.py", line 58, in __call__ Aug 07 17:03:06.410473 np0038147114 [email protected][56399]: ERROR keystone response = req.get_response(self.application) Aug 07 17:03:06.410473 np0038147114 [email protected][56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/request.py", line 1313, in send Aug 07 17:03:06.410473 np0038147114 [email protected][56399]: ERROR keystone status, headers, app_iter = self.call_application( Aug 07 17:03:06.410473 np0038147114 [email protected][56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/request.py", line 1278, in call_application Aug 07 17:03:06.410473 np0038147114 [email protected][56399]: ERROR keystone app_iter = application(self.environ, start_response) Aug 07 17:03:06.410473 np0038147114 [email protected][56399]: ERROR keystone File "/opt/stack/keystone/keystone/server/flask/request_processing/middleware/url_normalize.py", line 38, in __call__ Aug 07 17:03:06.410473 np0038147114 [email protected][56399]: ERROR keystone return self.app(environ, start_response) Aug 07 17:03:06.410473 np0038147114 [email protected][56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py", line 129, in __call__ Aug 07 17:03:06.410473 np0038147114 [email protected][56399]: ERROR keystone resp = self.call_func(req, *args, **kw) Aug 07 17:03:06.410473 np0038147114 [email protected][56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py", line 193, in call_func Aug 07 17:03:06.410473 np0038147114 [email protected][56399]: ERROR keystone return self.func(req, *args, **kwargs) Aug 07 17:03:06.410473 np0038147114 [email protected][56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/keystonemiddleware/auth_token/__init__.py", line 340, in __call__ Aug 07 17:03:06.410473 np0038147114 [email protected][56399]: ERROR keystone response = req.get_response(self._app) Aug 07 17:03:06.410473 np0038147114 [email protected][56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/request.py", line 1313, in send Aug 07 17:03:06.410473 np0038147114 [email protected][56399]: ERROR keystone status, headers, app_iter = self.call_application( Aug 07 17:03:06.411332 np0038147114 [email protected][56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/request.py", line 1278, in call_application Aug 07 17:03:06.411332 np0038147114 [email protected][56399]: ERROR keystone app_iter = application(self.environ, start_response) Aug 07 17:03:06.411332 np0038147114 [email protected][56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/werkzeug/middleware/dispatcher.py", line 81, in __call__ Aug 07 17:03:06.411332 np0038147114 [email protected][56399]: ERROR keystone return app(environ, start_response) Aug 07 17:03:06.411332 np0038147114 [email protected][56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask/app.py", line 1476, in wsgi_app Aug 07 17:03:06.411332 np0038147114 [email protected][56399]: ERROR keystone response = self.handle_exception(e) Aug 07 17:03:06.411332 np0038147114 [email protected][56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 298, in error_router Aug 07 17:03:06.411332 np0038147114 [email protected][56399]: ERROR keystone return original_handler(e) Aug 07 17:03:06.411332 np0038147114 [email protected][56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 298, in error_router Aug 07 17:03:06.411332 np0038147114 [email protected][56399]: ERROR keystone return original_handler(e) Aug 07 17:03:06.411332 np0038147114 [email protected][56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 298, in error_router Aug 07 17:03:06.411332 np0038147114 [email protected][56399]: ERROR keystone return original_handler(e) Aug 07 17:03:06.411332 np0038147114 [email protected][56399]: ERROR keystone [Previous line repeated 28 more times] Aug 07 17:03:06.411332 np0038147114 [email protected][56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask/app.py", line 1473, in wsgi_app Aug 07 17:03:06.411332 np0038147114 [email protected][56399]: ERROR keystone response = self.full_dispatch_request() Aug 07 17:03:06.411332 np0038147114 [email protected][56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask/app.py", line 882, in full_dispatch_request Aug 07 17:03:06.411332 np0038147114 [email protected][56399]: ERROR keystone rv = self.handle_user_exception(e) Aug 07 17:03:06.411332 np0038147114 [email protected][56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 298, in error_router Aug 07 17:03:06.411332 np0038147114 [email protected][56399]: ERROR keystone return original_handler(e) Aug 07 17:03:06.411332 np0038147114 [email protected][56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 298, in error_router Aug 07 17:03:06.412219 np0038147114 [email protected][56399]: ERROR keystone return original_handler(e) Aug 07 17:03:06.412219 np0038147114 [email protected][56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 298, in error_router Aug 07 17:03:06.412219 np0038147114 [email protected][56399]: ERROR keystone return original_handler(e) Aug 07 17:03:06.412219 np0038147114 [email protected][56399]: ERROR keystone [Previous line repeated 28 more times] Aug 07 17:03:06.412219 np0038147114 [email protected][56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask/app.py", line 880, in full_dispatch_request Aug 07 17:03:06.412219 np0038147114 [email protected][56399]: ERROR keystone rv = self.dispatch_request() Aug 07 17:03:06.412219 np0038147114 [email protected][56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask/app.py", line 865, in dispatch_request Aug 07 17:03:06.412219 np0038147114 [email protected][56399]: ERROR keystone return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) # type: ignore[no-any-return] Aug 07 17:03:06.412219 np0038147114 [email protected][56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 489, in wrapper Aug 07 17:03:06.412219 np0038147114 [email protected][56399]: ERROR keystone resp = resource(*args, **kwargs) Aug 07 17:03:06.412219 np0038147114 [email protected][56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask/views.py", line 110, in view Aug 07 17:03:06.412219 np0038147114 [email protected][56399]: ERROR keystone return current_app.ensure_sync(self.dispatch_request)(**kwargs) # type: ignore[no-any-return] Aug 07 17:03:06.412219 np0038147114 [email protected][56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 604, in dispatch_request Aug 07 17:03:06.412219 np0038147114 [email protected][56399]: ERROR keystone resp = meth(*args, **kwargs) Aug 07 17:03:06.412219 np0038147114 [email protected][56399]: ERROR keystone File "/opt/stack/keystone/keystone/server/flask/common.py", line 1146, in wrapper Aug 07 17:03:06.412219 np0038147114 [email protected][56399]: ERROR keystone return f(*args, **kwargs) Aug 07 17:03:06.412219 np0038147114 [email protected][56399]: ERROR keystone File "/opt/stack/keystone/keystone/api/auth.py", line 328, in post Aug 07 17:03:06.412219 np0038147114 [email protected][56399]: ERROR keystone token = authentication.authenticate_for_token(auth_data) Aug 07 17:03:06.412219 np0038147114 [email protected][56399]: ERROR keystone File "/opt/stack/keystone/keystone/api/_shared/authentication.py", line 202, in authenticate_for_token Aug 07 17:03:06.412219 np0038147114 [email protected][56399]: ERROR keystone authenticate(auth_info, auth_context) Aug 07 17:03:06.413072 np0038147114 [email protected][56399]: ERROR keystone File "/opt/stack/keystone/keystone/api/_shared/authentication.py", line 170, in authenticate Aug 07 17:03:06.413072 np0038147114 [email protected][56399]: ERROR keystone resp = method.authenticate(auth_info.get_method_data(method_name)) Aug 07 17:03:06.413072 np0038147114 [email protected][56399]: ERROR keystone File "/opt/stack/keystone/keystone/auth/plugins/password.py", line 34, in authenticate Aug 07 17:03:06.413072 np0038147114 [email protected][56399]: ERROR keystone PROVIDERS.identity_api.authenticate( Aug 07 17:03:06.413072 np0038147114 [email protected][56399]: ERROR keystone File "/opt/stack/keystone/keystone/common/manager.py", line 111, in wrapped Aug 07 17:03:06.413072 np0038147114 [email protected][56399]: ERROR keystone __ret_val = __f(*args, **kwargs) Aug 07 17:03:06.413072 np0038147114 [email protected][56399]: ERROR keystone File "/opt/stack/keystone/keystone/notifications.py", line 718, in wrapper Aug 07 17:03:06.413072 np0038147114 [email protected][56399]: ERROR keystone result = f(wrapped_self, user_id, *args, **kwargs) Aug 07 17:03:06.413072 np0038147114 [email protected][56399]: ERROR keystone File "/opt/stack/keystone/keystone/identity/core.py", line 533, in wrapper Aug 07 17:03:06.413072 np0038147114 [email protected][56399]: ERROR keystone return f(self, *args, **kwargs) Aug 07 17:03:06.413072 np0038147114 [email protected][56399]: ERROR keystone File "/opt/stack/keystone/keystone/identity/core.py", line 545, in wrapper Aug 07 17:03:06.413072 np0038147114 [email protected][56399]: ERROR keystone return f(self, *args, **kwargs) Aug 07 17:03:06.413072 np0038147114 [email protected][56399]: ERROR keystone File "/opt/stack/keystone/keystone/identity/core.py", line 1069, in authenticate Aug 07 17:03:06.413072 np0038147114 [email protected][56399]: ERROR keystone PROVIDERS.shadow_users_api.set_last_active_at(ref['id']) Aug 07 17:03:06.413072 np0038147114 [email protected][56399]: ERROR keystone File "/opt/stack/keystone/keystone/identity/shadow_backends/sql.py", line 166, in set_last_active_at Aug 07 17:03:06.413072 np0038147114 [email protected][56399]: ERROR keystone with sql.session_for_write() as session: Aug 07 17:03:06.413072 np0038147114 [email protected][56399]: ERROR keystone File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ Aug 07 17:03:06.413072 np0038147114 [email protected][56399]: ERROR keystone next(self.gen) Aug 07 17:03:06.413072 np0038147114 [email protected][56399]: ERROR keystone File "/opt/stack/osprofiler/osprofiler/sqlalchemy.py", line 56, in wrap_session Aug 07 17:03:06.413072 np0038147114 [email protected][56399]: ERROR keystone with sess as s: Aug 07 17:03:06.413072 np0038147114 [email protected][56399]: ERROR keystone File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ Aug 07 17:03:06.413072 np0038147114 [email protected][56399]: ERROR keystone next(self.gen) Aug 07 17:03:06.414043 np0038147114 [email protected][56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1042, in _transaction_scope Aug 07 17:03:06.414043 np0038147114 [email protected][56399]: ERROR keystone with current._produce_block( Aug 07 17:03:06.414043 np0038147114 [email protected][56399]: ERROR keystone File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ Aug 07 17:03:06.414043 np0038147114 [email protected][56399]: ERROR keystone next(self.gen) Aug 07 17:03:06.414043 np0038147114 [email protected][56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 649, in _session Aug 07 17:03:06.414043 np0038147114 [email protected][56399]: ERROR keystone with excutils.save_and_reraise_exception(): Aug 07 17:03:06.414043 np0038147114 [email protected][56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__ Aug 07 17:03:06.414043 np0038147114 [email protected][56399]: ERROR keystone self.force_reraise() Aug 07 17:03:06.414043 np0038147114 [email protected][56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise Aug 07 17:03:06.414043 np0038147114 [email protected][56399]: ERROR keystone raise self.value Aug 07 17:03:06.414043 np0038147114 [email protected][56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 647, in _session Aug 07 17:03:06.414043 np0038147114 [email protected][56399]: ERROR keystone self._end_session_transaction(self.session) Aug 07 17:03:06.414043 np0038147114 [email protected][56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 675, in _end_session_transaction Aug 07 17:03:06.414043 np0038147114 [email protected][56399]: ERROR keystone session.commit() Aug 07 17:03:06.414043 np0038147114 [email protected][56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 2017, in commit Aug 07 17:03:06.414043 np0038147114 [email protected][56399]: ERROR keystone trans.commit(_to_root=True) Aug 07 17:03:06.414043 np0038147114 [email protected][56399]: ERROR keystone File "<string>", line 2, in commit Aug 07 17:03:06.414043 np0038147114 [email protected][56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/state_changes.py", line 139, in _go Aug 07 17:03:06.414043 np0038147114 [email protected][56399]: ERROR keystone ret_value = fn(self, *arg, **kw) Aug 07 17:03:06.414043 np0038147114 [email protected][56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 1302, in commit Aug 07 17:03:06.414043 np0038147114 [email protected][56399]: ERROR keystone self._prepare_impl() Aug 07 17:03:06.414927 np0038147114 [email protected][56399]: ERROR keystone File "<string>", line 2, in _prepare_impl Aug 07 17:03:06.414927 np0038147114 [email protected][56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/state_changes.py", line 139, in _go Aug 07 17:03:06.414927 np0038147114 [email protected][56399]: ERROR keystone ret_value = fn(self, *arg, **kw) Aug 07 17:03:06.414927 np0038147114 [email protected][56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 1277, in _prepare_impl Aug 07 17:03:06.414927 np0038147114 [email protected][56399]: ERROR keystone self.session.flush() Aug 07 17:03:06.414927 np0038147114 [email protected][56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 4341, in flush Aug 07 17:03:06.414927 np0038147114 [email protected][56399]: ERROR keystone self._flush(objects) Aug 07 17:03:06.414927 np0038147114 [email protected][56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 4476, in _flush Aug 07 17:03:06.414927 np0038147114 [email protected][56399]: ERROR keystone with util.safe_reraise(): Aug 07 17:03:06.414927 np0038147114 [email protected][56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 146, in __exit__ Aug 07 17:03:06.414927 np0038147114 [email protected][56399]: ERROR keystone raise exc_value.with_traceback(exc_tb) Aug 07 17:03:06.414927 np0038147114 [email protected][56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 4437, in _flush Aug 07 17:03:06.414927 np0038147114 [email protected][56399]: ERROR keystone flush_context.execute() Aug 07 17:03:06.414927 np0038147114 [email protected][56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/unitofwork.py", line 466, in execute Aug 07 17:03:06.414927 np0038147114 [email protected][56399]: ERROR keystone rec.execute(self) Aug 07 17:03:06.414927 np0038147114 [email protected][56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/unitofwork.py", line 642, in execute Aug 07 17:03:06.414927 np0038147114 [email protected][56399]: ERROR keystone util.preloaded.orm_persistence.save_obj( Aug 07 17:03:06.414927 np0038147114 [email protected][56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/persistence.py", line 85, in save_obj Aug 07 17:03:06.414927 np0038147114 [email protected][56399]: ERROR keystone _emit_update_statements( Aug 07 17:03:06.414927 np0038147114 [email protected][56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/persistence.py", line 912, in _emit_update_statements Aug 07 17:03:06.415874 np0038147114 [email protected][56399]: ERROR keystone c = connection.execute( Aug 07 17:03:06.415874 np0038147114 [email protected][56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1418, in execute Aug 07 17:03:06.415874 np0038147114 [email protected][56399]: ERROR keystone return meth( Aug 07 17:03:06.415874 np0038147114 [email protected][56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/sql/elements.py", line 515, in _execute_on_connection Aug 07 17:03:06.415874 np0038147114 [email protected][56399]: ERROR keystone return connection._execute_clauseelement( Aug 07 17:03:06.415874 np0038147114 [email protected][56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1640, in _execute_clauseelement Aug 07 17:03:06.415874 np0038147114 [email protected][56399]: ERROR keystone ret = self._execute_context( Aug 07 17:03:06.415874 np0038147114 [email protected][56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1846, in _execute_context Aug 07 17:03:06.415874 np0038147114 [email protected][56399]: ERROR keystone return self._exec_single_context( Aug 07 17:03:06.415874 np0038147114 [email protected][56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1889, in _exec_single_context Aug 07 17:03:06.415874 np0038147114 [email protected][56399]: ERROR keystone str_statement, effective_parameters = fn( Aug 07 17:03:06.415874 np0038147114 [email protected][56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/engine/events.py", line 176, in wrap_before_cursor_execute Aug 07 17:03:06.415874 np0038147114 [email protected][56399]: ERROR keystone orig_fn( Aug 07 17:03:06.415874 np0038147114 [email protected][56399]: ERROR keystone File "/opt/stack/osprofiler/osprofiler/sqlalchemy.py", line 71, in handler Aug 07 17:03:06.415874 np0038147114 [email protected][56399]: ERROR keystone profiler.start(name, info=info) Aug 07 17:03:06.415874 np0038147114 [email protected][56399]: ERROR keystone File "/opt/stack/osprofiler/osprofiler/profiler.py", line 82, in start Aug 07 17:03:06.415874 np0038147114 [email protected][56399]: ERROR keystone profiler.start(name, info=info) Aug 07 17:03:06.415874 np0038147114 [email protected][56399]: ERROR keystone File "/opt/stack/osprofiler/osprofiler/profiler.py", line 410, in start Aug 07 17:03:06.415874 np0038147114 [email protected][56399]: ERROR keystone self._notify("%s-start" % name, info) Aug 07 17:03:06.415874 np0038147114 [email protected][56399]: ERROR keystone File "/opt/stack/osprofiler/osprofiler/profiler.py", line 436, in _notify Aug 07 17:03:06.415874 np0038147114 [email protected][56399]: ERROR keystone notifier.notify(payload) Aug 07 17:03:06.416936 np0038147114 [email protected][56399]: ERROR keystone File "/opt/stack/osprofiler/osprofiler/notifier.py", line 38, in notify Aug 07 17:03:06.416936 np0038147114 [email protected][56399]: ERROR keystone __notifier(info) Aug 07 17:03:06.416936 np0038147114 [email protected][56399]: ERROR keystone File "/opt/stack/osprofiler/osprofiler/drivers/redis_driver.py", line 75, in notify Aug 07 17:03:06.416936 np0038147114 [email protected][56399]: ERROR keystone self.db.lpush(key, jsonutils.dumps(data)) Aug 07 17:03:06.416936 np0038147114 [email protected][56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_serialization/jsonutils.py", line 198, in dumps Aug 07 17:03:06.416936 np0038147114 [email protected][56399]: ERROR keystone return json.dumps(obj, default=default, **kwargs) Aug 07 17:03:06.416936 np0038147114 [email protected][56399]: ERROR keystone File "/usr/lib/python3.10/json/__init__.py", line 238, in dumps Aug 07 17:03:06.416936 np0038147114 [email protected][56399]: ERROR keystone **kw).encode(obj) Aug 07 17:03:06.416936 np0038147114 [email protected][56399]: ERROR keystone File "/usr/lib/python3.10/json/encoder.py", line 199, in encode Aug 07 17:03:06.416936 np0038147114 [email protected][56399]: ERROR keystone chunks = self.iterencode(o, _one_shot=True) Aug 07 17:03:06.416936 np0038147114 [email protected][56399]: ERROR keystone File "/usr/lib/python3.10/json/encoder.py", line 257, in iterencode Aug 07 17:03:06.416936 np0038147114 [email protected][56399]: ERROR keystone return _iterencode(o, 0) Aug 07 17:03:06.416936 np0038147114 [email protected][56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_serialization/jsonutils.py", line 176, in to_primitive Aug 07 17:03:06.416936 np0038147114 [email protected][56399]: ERROR keystone raise ValueError("Cannot convert %r to primitive" % (value,)) Aug 07 17:03:06.416936 np0038147114 [email protected][56399]: ERROR keystone ValueError: Cannot convert datetime.date(2024, 8, 7) to primitive Aug 07 17:03:06.416936 np0038147114 [email protected][56399]: ERROR keystone Aug 07 17:03:06.416936 np0038147114 [email protected][56399]: [pid: 56399|app: 0|req: 225/451] 200.225.47.15 () {70 vars in 1374 bytes} [Wed Aug 7 17:03:06 2024] POST /identity/v3/auth/tokens => generated 0 bytes in 61 msecs (HTTP/1.1 500) 0 headers in 0 bytes (0 switches on core 0) Example failure:- https://2a378a06b97cbac17818-b80014e5a7a6453c822c4dc8f22159da.ssl.cf5.rackcdn.com/925376/2/check/neutron- ovs-rally-task/fb03014/job-output.txt also cinder-rally-tasks broken Builds:- https://zuul.opendev.org/t/openstack/builds?job_name=neutron- ovs-rally-task&job_name=cinder-rally-task&branch=master&skip=0 Broken since patch https://review.opendev.org/c/openstack/keystone/+/924892 To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/2076430/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
