yes, via ceph orch.
---
service_type: rgw
service_id: eu-central-lz
service_name: rgw.eu-central-lz
placement:
count_per_host: 1
label: rgw
spec:
config:
debug_rgw: 0
rgw_dns_name: s3.eu-central-lz.tld
rgw_dns_s3website_name: s3-website.eu-central-lz.tld
rgw_keystone_token_cache_size: 100000
rgw_thread_pool_size: 512
rgw_frontend_port: 7480
rgw_frontend_type: beast
rgw_realm: ovh
rgw_zone: eu-central-lz
rgw_zonegroup: eu-central-lz
Am Mo., 3. Nov. 2025 um 17:09 Uhr schrieb Anthony D'Atri <
[email protected]>:
> How is your RGW service deployed? ceph orch? Something else?
>
> On Nov 3, 2025, at 10:56 AM, Boris <[email protected]> wrote:
>
> Hi Anthony,
> here are the config values we've set or with their defaults. There is
> no rgw_keystone_token_cache_ttl (neither in the documentation, nor can I
> set it via ceph config set client.rgw rgw_keystone_token_cache_ttl 3600):
>
> ~# ceph config show-with-defaults rgw.rgw1 | grep rgw_keystone | column -t
> rgw_keystone_accepted_admin_roles default
>
> rgw_keystone_accepted_roles objectstore_operator
> mon
> rgw_keystone_admin_domain default
> mon
> rgw_keystone_admin_password yyyyyyyy
> mon
> rgw_keystone_admin_password_path default
>
> rgw_keystone_admin_project services
> mon
> rgw_keystone_admin_tenant default
>
> rgw_keystone_admin_token default
>
> rgw_keystone_admin_token_path default
>
> rgw_keystone_admin_user xxxxxxx
> mon
> rgw_keystone_api_version 3
> mon
> rgw_keystone_barbican_domain default
>
> rgw_keystone_barbican_password default
>
> rgw_keystone_barbican_project default
>
> rgw_keystone_barbican_tenant default
>
> rgw_keystone_barbican_user default
>
> rgw_keystone_expired_token_cache_expiration 3600
> default
> rgw_keystone_implicit_tenants false
> default
> rgw_keystone_service_token_accepted_roles admin
> default
> rgw_keystone_service_token_enabled false
> default
> rgw_keystone_token_cache_size 100000
> mon <-- i've set this to test if this solves the problem, but
> this is the default value
> rgw_keystone_url https://auth.tld
> mon
> rgw_keystone_verify_ssl true
> default
>
>
>
> Am Mo., 3. Nov. 2025 um 16:40 Uhr schrieb Anthony D'Atri <
> [email protected]>:
>
>> Check the values of rgw_keystone_token_cache_size and
>> rgw_keystone_token_cache_ttl and other rgw_keystone options.
>>
>> I've seen at least one deployment tool that disabled Keystone caching for
>> dev purposes, but leaked that into the release code, which deployed RGW
>> with Rook with a configmap override.
>>
>>
>> > On Nov 3, 2025, at 9:52 AM, Boris <[email protected]> wrote:
>> >
>> > Hi,
>> > I am currently debugging a problem that the radosgw keystone token cache
>> > seems not to work properly. Or at all. I tried to debug it and attached
>> the
>> > rgw_debug log set to 10. I've truncated to only show the part from "No
>> > stored secret string, cache miss" until the request is done.
>> >
>> > The failed request hits a rate limit on the keystone which currently
>> takes
>> > around 2k answered requests per minute.
>> > Any ideas what I did wrong?
>> >
>> > * All requests were done within 10 seconds and were only an ls to show
>> > buckets.
>> > * This particular RGW only took my requests during testing.
>> > * We didn't set any timeouts or special cache configs in ceph
>> > * system time is correct
>> >
>> >
>> > First request worked instantly:
>> >
>> > req 8122732607072897744 0.106001295s s3:list_buckets No stored secret
>> > string, cache miss
>> > [4.0K blob data]
>> > req 8122732607072897744 0.315003842s s3:list_buckets s3 keystone:
>> validated
>> > token: 8144848695793469:user-9XGYcbFNUVTQ expires: 1762266594
>> > req 8122732607072897744 0.315003842s s3:list_buckets cache get:
>> >
>> name=eu-central-lz.rgw.meta+users.uid+a13f0472be744104ad1f64bb2855cdee$a13f0472be744104ad1f64bb2855cdee
>> > : hit (negative entry)
>> > req 8122732607072897744 0.315003842s s3:list_buckets cache get:
>> > name=eu-central-lz.rgw.meta+users.uid+a13f0472be744104ad1f64bb2855cdee :
>> > hit (requested=0x13, cached=0x13)
>> > req 8122732607072897744 0.315003842s s3:list_buckets normalizing buckets
>> > and tenants
>> > req 8122732607072897744 0.315003842s s->object=<NULL> s->bucket=
>> > req 8122732607072897744 0.315003842s s3:list_buckets init permissions
>> > req 8122732607072897744 0.315003842s s3:list_buckets cache get:
>> > name=eu-central-lz.rgw.meta+users.uid+a13f0472be744104ad1f64bb2855cdee :
>> > hit (requested=0x13, cached=0x13)
>> > req 8122732607072897744 0.315003842s s3:list_buckets recalculating
>> target
>> > req 8122732607072897744 0.315003842s s3:list_buckets reading permissions
>> > req 8122732607072897744 0.315003842s s3:list_buckets init op
>> > req 8122732607072897744 0.315003842s s3:list_buckets verifying op mask
>> > req 8122732607072897744 0.315003842s s3:list_buckets verifying op
>> > permissions
>> > req 8122732607072897744 0.315003842s s3:list_buckets verifying op params
>> > req 8122732607072897744 0.315003842s s3:list_buckets pre-executing
>> > req 8122732607072897744 0.315003842s s3:list_buckets check rate limiting
>> > req 8122732607072897744 0.315003842s s3:list_buckets executing
>> > req 8122732607072897744 0.315003842s s3:list_buckets completing
>> > req 8122732607072897744 0.315003842s cache get:
>> > name=eu-central-lz.rgw.log++script.postrequest. : hit (negative entry)
>> > req 8122732607072897744 0.315003842s s3:list_buckets op status=0
>> > req 8122732607072897744 0.315003842s s3:list_buckets http status=200
>> > ====== req done req=0x74659e51b6f0 op status=0 http_status=200
>> > latency=0.315003842s ======
>> >
>> > 2nd request failed
>> >
>> > req 10422983006485317789 0.061000749s s3:list_buckets cache get:
>> > name=eu-central-lz.rgw.meta+users.keys+05917cf2ee9d4fdea8baf6a3348ca33a
>> :
>> > hit (negative entry)
>> > req 10422983006485317789 0.061000749s s3:list_buckets error reading user
>> > info, uid=05917cf2ee9d4fdea8baf6a3348ca33a can't authenticate
>> > req 10422983006485317789 0.061000749s s3:list_buckets Failed the auth
>> > strategy, reason=-5
>> > failed to authorize request
>> > WARNING: set_req_state_err err_no=5 resorting to 500
>> > req 10422983006485317789 0.061000749s cache get:
>> > name=eu-central-lz.rgw.log++script.postrequest. : hit (negative entry)
>> > req 10422983006485317789 0.061000749s s3:list_buckets op status=0
>> > req 10422983006485317789 0.061000749s s3:list_buckets http status=500
>> > ====== req done req=0x74659e51b6f0 op status=0 http_status=500
>> > latency=0.061000749s ======
>> >
>> > 3rd requests went through again
>> >
>> > req 13123970335019889535 0.000000000s s3:list_buckets No stored secret
>> > string, cache miss
>> > [250B blob data]
>> > req 13123970335019889535 0.204002500s s3:list_buckets s3 keystone:
>> > validated token: 8144848695793469:user-9XGYcbFNUVTQ expires: 1762266602
>> > req 13123970335019889535 0.204002500s s3:list_buckets cache get:
>> >
>> name=eu-central-lz.rgw.meta+users.uid+a13f0472be744104ad1f64bb2855cdee$a13f0472be744104ad1f64bb2855cdee
>> > : hit (negative entry)
>> > req 13123970335019889535 0.204002500s s3:list_buckets cache get:
>> > name=eu-central-lz.rgw.meta+users.uid+a13f0472be744104ad1f64bb2855cdee :
>> > hit (requested=0x13, cached=0x13)
>> > req 13123970335019889535 0.204002500s s3:list_buckets normalizing
>> buckets
>> > and tenants
>> > req 13123970335019889535 0.204002500s s->object=<NULL> s->bucket=
>> > req 13123970335019889535 0.204002500s s3:list_buckets init permissions
>> > req 13123970335019889535 0.204002500s s3:list_buckets cache get:
>> > name=eu-central-lz.rgw.meta+users.uid+a13f0472be744104ad1f64bb2855cdee :
>> > hit (requested=0x13, cached=0x13)
>> > req 13123970335019889535 0.204002500s s3:list_buckets recalculating
>> target
>> > req 13123970335019889535 0.204002500s s3:list_buckets reading
>> permissions
>> > req 13123970335019889535 0.204002500s s3:list_buckets init op
>> > req 13123970335019889535 0.204002500s s3:list_buckets verifying op mask
>> > req 13123970335019889535 0.204002500s s3:list_buckets verifying op
>> > permissions
>> > req 13123970335019889535 0.204002500s s3:list_buckets verifying op
>> params
>> > req 13123970335019889535 0.204002500s s3:list_buckets pre-executing
>> > req 13123970335019889535 0.204002500s s3:list_buckets check rate
>> limiting
>> > req 13123970335019889535 0.204002500s s3:list_buckets executing
>> > req 13123970335019889535 0.204002500s s3:list_buckets completing
>> > req 13123970335019889535 0.204002500s cache get:
>> > name=eu-central-lz.rgw.log++script.postrequest. : hit (negative entry)
>> > req 13123970335019889535 0.204002500s s3:list_buckets op status=0
>> > req 13123970335019889535 0.204002500s s3:list_buckets http status=200
>> > ====== req done req=0x74659e51b6f0 op status=0 http_status=200
>> > latency=0.204002500s ======
>> >
>> >
>> >
>> >
>> >
>> > --
>> > Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im
>> > groüen Saal.
>> > _______________________________________________
>> > ceph-users mailing list -- [email protected]
>> > To unsubscribe send an email to [email protected]
>>
>>
>
> --
> Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im
> groüen Saal.
>
>
>
--
Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im
groüen Saal.
_______________________________________________
ceph-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
