Hi Tobias, I just pumped up the rgw_debug to 20 and generated new output: https://pastebin.com/PcSUSWGY I hope that I redacted all the sensitive data. :)
3 Requests to list all my buckets in <10 seconds. The 1st request showd me my buckets, then 2nd requests resulted in a 500 error and thew 3rd showed me my buckets again. For me this currently looks like I get a "429 Too Many Requests" from the keystone on all the three requests that I made and I would have expected to see this error only on the 2nd requests. Weird is also line 104-109. I have no idea how the content of the /etc/hosts file made it into the log. The keystone user that we have in the "rgw_keystone_admin_user" is not a keystone admin. The people that maintain the keystone just told me "The user doesn't have admin and we would not grant it." The "rgw_s3_auth_order" is default. We didn't touch it. "sts, external, local" Am Mi., 5. Nov. 2025 um 16:32 Uhr schrieb Tobias Urdin - Binero < [email protected]>: > Hello Boris, > > What roles is assigned to the Keystone user configured in > rgw_keystone_admin_user? It needs the > admin role in order to be allowed the > /v3/users/<user_id>/credentials/OS-EC2/<access_key> API request. > > openstack role assignment list —names —user <rgw_keystone_admin_user > value> > > A part from that I don’t understand the “2nd request failed” part as that > seems to be from the LocalEngine > and is not related to Keystone, if you have the default value > for rgw_s3_auth_order the only thing I can > think off is that there is a bug or you’re missing some patch like [1] [2] > but that’s just a guess. > > /Tobias > > [1] https://github.com/ceph/ceph/pull/53846 > [2] https://github.com/ceph/ceph/pull/53680 > > > On 4 Nov 2025, at 11:32, Boris <[email protected]> wrote: > > I've created an upstream ticket > https://eur05.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftracker.ceph.com%2Fissues%2F73709&data=05%7C02%7Ctobias.urdin%40binero.com%7C17fa249f3ee94151d37a08de1b8d8e9f%7C89d97f28180f459da0e585855aa63f6c%7C0%7C0%7C638978491958438817%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=MVedgbK0xyCFJY%2FuA%2FskKoY1VwBv6ikMrfVCjT9f%2Bro%3D&reserved=0 > <https://tracker.ceph.com/issues/73709> > > Am Mo., 3. Nov. 2025 um 17:13 Uhr schrieb Boris <[email protected]>: > > yes, via ceph orch. > > --- > service_type: rgw > service_id: eu-central-lz > service_name: rgw.eu-central-lz > placement: > count_per_host: 1 > label: rgw > spec: > config: > debug_rgw: 0 > rgw_dns_name: s3.eu-central-lz.tld > rgw_dns_s3website_name: s3-website.eu-central-lz.tld > rgw_keystone_token_cache_size: 100000 > rgw_thread_pool_size: 512 > rgw_frontend_port: 7480 > rgw_frontend_type: beast > rgw_realm: ovh > rgw_zone: eu-central-lz > rgw_zonegroup: eu-central-lz > > Am Mo., 3. Nov. 2025 um 17:09 Uhr schrieb Anthony D'Atri < > [email protected]>: > > How is your RGW service deployed? ceph orch? Something else? > > On Nov 3, 2025, at 10:56 AM, Boris <[email protected]> wrote: > > Hi Anthony, > here are the config values we've set or with their defaults. There is > no rgw_keystone_token_cache_ttl (neither in the documentation, nor can I > set it via ceph config set client.rgw rgw_keystone_token_cache_ttl 3600): > > ~# ceph config show-with-defaults rgw.rgw1 | grep rgw_keystone | column -t > rgw_keystone_accepted_admin_roles default > > rgw_keystone_accepted_roles objectstore_operator > mon > rgw_keystone_admin_domain default > mon > rgw_keystone_admin_password yyyyyyyy > mon > rgw_keystone_admin_password_path default > > rgw_keystone_admin_project services > mon > rgw_keystone_admin_tenant default > > rgw_keystone_admin_token default > > rgw_keystone_admin_token_path default > > rgw_keystone_admin_user xxxxxxx > mon > rgw_keystone_api_version 3 > mon > rgw_keystone_barbican_domain default > > rgw_keystone_barbican_password default > > rgw_keystone_barbican_project default > > rgw_keystone_barbican_tenant default > > rgw_keystone_barbican_user default > > rgw_keystone_expired_token_cache_expiration 3600 > default > rgw_keystone_implicit_tenants false > default > rgw_keystone_service_token_accepted_roles admin > default > rgw_keystone_service_token_enabled false > default > rgw_keystone_token_cache_size 100000 > mon <-- i've set this to test if this solves the problem, but > this is the default value > rgw_keystone_url > https://eur05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fauth.tld%2F&data=05%7C02%7Ctobias.urdin%40binero.com%7C17fa249f3ee94151d37a08de1b8d8e9f%7C89d97f28180f459da0e585855aa63f6c%7C0%7C0%7C638978491958459086%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=jNFs4KDcToTVmzLnSEg0lgDIKHLtO6yt5zYTE6fpWao%3D&reserved=0 > <https://auth.tld/> > mon > rgw_keystone_verify_ssl true > default > > > > Am Mo., 3. Nov. 2025 um 16:40 Uhr schrieb Anthony D'Atri < > [email protected]>: > > Check the values of rgw_keystone_token_cache_size and > rgw_keystone_token_cache_ttl and other rgw_keystone options. > > I've seen at least one deployment tool that disabled Keystone caching > for dev purposes, but leaked that into the release code, which deployed RGW > with Rook with a configmap override. > > > On Nov 3, 2025, at 9:52 AM, Boris <[email protected]> wrote: > > Hi, > I am currently debugging a problem that the radosgw keystone token > > cache > > seems not to work properly. Or at all. I tried to debug it and > > attached the > > rgw_debug log set to 10. I've truncated to only show the part from "No > stored secret string, cache miss" until the request is done. > > The failed request hits a rate limit on the keystone which currently > > takes > > around 2k answered requests per minute. > Any ideas what I did wrong? > > * All requests were done within 10 seconds and were only an ls to show > buckets. > * This particular RGW only took my requests during testing. > * We didn't set any timeouts or special cache configs in ceph > * system time is correct > > > First request worked instantly: > > req 8122732607072897744 0.106001295s s3:list_buckets No stored secret > string, cache miss > [4.0K blob data] > req 8122732607072897744 0.315003842s s3:list_buckets s3 keystone: > > validated > > token: 8144848695793469:user-9XGYcbFNUVTQ expires: 1762266594 > req 8122732607072897744 0.315003842s s3:list_buckets cache get: > > > name=eu-central-lz.rgw.meta+users.uid+a13f0472be744104ad1f64bb2855cdee$a13f0472be744104ad1f64bb2855cdee > > : hit (negative entry) > req 8122732607072897744 0.315003842s s3:list_buckets cache get: > name=eu-central-lz.rgw.meta+users.uid+a13f0472be744104ad1f64bb2855cdee > > : > > hit (requested=0x13, cached=0x13) > req 8122732607072897744 0.315003842s s3:list_buckets normalizing > > buckets > > and tenants > req 8122732607072897744 0.315003842s s->object=<NULL> s->bucket= > req 8122732607072897744 0.315003842s s3:list_buckets init permissions > req 8122732607072897744 0.315003842s s3:list_buckets cache get: > name=eu-central-lz.rgw.meta+users.uid+a13f0472be744104ad1f64bb2855cdee > > : > > hit (requested=0x13, cached=0x13) > req 8122732607072897744 0.315003842s s3:list_buckets recalculating > > target > > req 8122732607072897744 0.315003842s s3:list_buckets reading > > permissions > > req 8122732607072897744 0.315003842s s3:list_buckets init op > req 8122732607072897744 0.315003842s s3:list_buckets verifying op mask > req 8122732607072897744 0.315003842s s3:list_buckets verifying op > permissions > req 8122732607072897744 0.315003842s s3:list_buckets verifying op > > params > > req 8122732607072897744 0.315003842s s3:list_buckets pre-executing > req 8122732607072897744 0.315003842s s3:list_buckets check rate > > limiting > > req 8122732607072897744 0.315003842s s3:list_buckets executing > req 8122732607072897744 0.315003842s s3:list_buckets completing > req 8122732607072897744 0.315003842s cache get: > name=eu-central-lz.rgw.log++script.postrequest. : hit (negative entry) > req 8122732607072897744 0.315003842s s3:list_buckets op status=0 > req 8122732607072897744 0.315003842s s3:list_buckets http status=200 > ====== req done req=0x74659e51b6f0 op status=0 http_status=200 > latency=0.315003842s ====== > > 2nd request failed > > req 10422983006485317789 0.061000749s s3:list_buckets cache get: > > name=eu-central-lz.rgw.meta+users.keys+05917cf2ee9d4fdea8baf6a3348ca33a : > > hit (negative entry) > req 10422983006485317789 0.061000749s s3:list_buckets error reading > > user > > info, uid=05917cf2ee9d4fdea8baf6a3348ca33a can't authenticate > req 10422983006485317789 0.061000749s s3:list_buckets Failed the auth > strategy, reason=-5 > failed to authorize request > WARNING: set_req_state_err err_no=5 resorting to 500 > req 10422983006485317789 0.061000749s cache get: > name=eu-central-lz.rgw.log++script.postrequest. : hit (negative entry) > req 10422983006485317789 0.061000749s s3:list_buckets op status=0 > req 10422983006485317789 0.061000749s s3:list_buckets http status=500 > ====== req done req=0x74659e51b6f0 op status=0 http_status=500 > latency=0.061000749s ====== > > 3rd requests went through again > > req 13123970335019889535 0.000000000s s3:list_buckets No stored secret > string, cache miss > [250B blob data] > req 13123970335019889535 0.204002500s s3:list_buckets s3 keystone: > validated token: 8144848695793469:user-9XGYcbFNUVTQ expires: 1762266602 > req 13123970335019889535 0.204002500s s3:list_buckets cache get: > > > name=eu-central-lz.rgw.meta+users.uid+a13f0472be744104ad1f64bb2855cdee$a13f0472be744104ad1f64bb2855cdee > > : hit (negative entry) > req 13123970335019889535 0.204002500s s3:list_buckets cache get: > name=eu-central-lz.rgw.meta+users.uid+a13f0472be744104ad1f64bb2855cdee > > : > > hit (requested=0x13, cached=0x13) > req 13123970335019889535 0.204002500s s3:list_buckets normalizing > > buckets > > and tenants > req 13123970335019889535 0.204002500s s->object=<NULL> s->bucket= > req 13123970335019889535 0.204002500s s3:list_buckets init permissions > req 13123970335019889535 0.204002500s s3:list_buckets cache get: > name=eu-central-lz.rgw.meta+users.uid+a13f0472be744104ad1f64bb2855cdee > > : > > hit (requested=0x13, cached=0x13) > req 13123970335019889535 0.204002500s s3:list_buckets recalculating > > target > > req 13123970335019889535 0.204002500s s3:list_buckets reading > > permissions > > req 13123970335019889535 0.204002500s s3:list_buckets init op > req 13123970335019889535 0.204002500s s3:list_buckets verifying op mask > req 13123970335019889535 0.204002500s s3:list_buckets verifying op > permissions > req 13123970335019889535 0.204002500s s3:list_buckets verifying op > > params > > req 13123970335019889535 0.204002500s s3:list_buckets pre-executing > req 13123970335019889535 0.204002500s s3:list_buckets check rate > > limiting > > req 13123970335019889535 0.204002500s s3:list_buckets executing > req 13123970335019889535 0.204002500s s3:list_buckets completing > req 13123970335019889535 0.204002500s cache get: > name=eu-central-lz.rgw.log++script.postrequest. : hit (negative entry) > req 13123970335019889535 0.204002500s s3:list_buckets op status=0 > req 13123970335019889535 0.204002500s s3:list_buckets http status=200 > ====== req done req=0x74659e51b6f0 op status=0 http_status=200 > latency=0.204002500s ====== > > > > > > -- > Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend > > im > > groüen Saal. > _______________________________________________ > ceph-users mailing list -- [email protected] > To unsubscribe send an email to [email protected] > > > > > -- > Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im > groüen Saal. > > > > > -- > Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im > groüen Saal. > > > > -- > Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im > groüen Saal. > _______________________________________________ > ceph-users mailing list -- [email protected] > To unsubscribe send an email to [email protected] > > > -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal. _______________________________________________ ceph-users mailing list -- [email protected] To unsubscribe send an email to [email protected]
