Hi Folks,
Looking for some advice on RGW service specs and Cephadm. I've read the docs
here:
RGW Service — Ceph
Documentation<https://docs.ceph.com/en/reef/cephadm/services/rgw/>
Using a service spec I can deploy a RGW:
service_type: rgw
service_id: '25069123'
service_name: rgw.25069123
placement:
hosts:
- raynor-sc-1
extra_container_args:
- -v
- /etc/pki:/etc/pki:ro
- -v
- /etc/ssl/:/etc/ssl:ro
spec:
rgw_frontend_port: 7480
rgw_realm: geored_realm
rgw_zone: siteA
rgw_zonegroup: geored_zg
ssl: true
rgw_frontend_extra_args:
- "ssl_certificate=/etc/ssl/certs/server.crt"
- "ssl_private_key=/etc/ssl/private/server.key"
But it fails to start because my "rgw_frontends" config ends up as this. Notice
the two entries of "ssl_certificate".
beast ssl_port=7480 ssl_certificate=config://rgw/cert/rgw.25069123
ssl_certificate=/etc/ssl/certs/server.crt
ssl_private_key=/etc/ssl/private/server.key
That causes it to fail to start:
debug 2025-01-16T15:26:54.219+0000 7f65bcfdc840 0 deferred set uid:gid to
167:167 (ceph:ceph)
debug 2025-01-16T15:26:54.219+0000 7f65bcfdc840 0 ceph version 19.2.0
(16063ff2022298c9300e49a547a16ffda59baf13) squid (stable), process radosgw, pid
7
debug 2025-01-16T15:26:54.219+0000 7f65bcfdc840 0 framework: beast
debug 2025-01-16T15:26:54.219+0000 7f65bcfdc840 0 framework conf key:
ssl_port, val: 7480
debug 2025-01-16T15:26:54.219+0000 7f65bcfdc840 0 framework conf key:
ssl_certificate, val: config://rgw/cert/rgw.25069123
debug 2025-01-16T15:26:54.219+0000 7f65bcfdc840 0 framework conf key:
ssl_certificate, val: /etc/ssl/certs/server.crt
debug 2025-01-16T15:26:54.219+0000 7f65bcfdc840 0 framework conf key:
ssl_private_key, val: /etc/ssl/private/server.key
debug 2025-01-16T15:26:54.520+0000 7f65bcfdc840 -1 LDAP not started since no
server URIs were provided in the configuration.
debug 2025-01-16T15:26:54.531+0000 7f65bcfdc840 0 framework: beast
debug 2025-01-16T15:26:54.531+0000 7f65bcfdc840 0 framework conf key:
ssl_certificate, val: config://rgw/cert/$realm/$zone.crt
debug 2025-01-16T15:26:54.531+0000 7f65bcfdc840 0 framework conf key:
ssl_private_key, val: config://rgw/cert/$realm/$zone.key
debug 2025-01-16T15:26:54.597+0000 7f65bcfdc840 0 starting handler: beast
debug 2025-01-16T15:26:54.601+0000 7f65bcfdc840 -1 ssl_certificate was not
found: rgw/cert/rgw.25069123
debug 2025-01-16T15:26:54.602+0000 7f65bcfdc840 -1 no ssl_certificate
configured for ssl_port
debug 2025-01-16T15:26:54.602+0000 7f65bcfdc840 -1 ERROR: failed initializing
frontend
debug 2025-01-16T15:26:54.602+0000 7f65bcfdc840 -1 ERROR: initialize frontend
fail, r = 22
Correcting that config manually:
$ ceph config set client.rgw.25069123.raynor-sc-1.qiatgr rgw_frontends
"beast ssl_port=7480 ssl_certificate=/etc/ssl/certs/server.crt
ssl_private_key=/etc/ssl/private/server.key"
Then allows the RGW to start:
debug 2025-01-16T15:28:17.391+0000 7fba6e85a840 0 deferred set uid:gid to
167:167 (ceph:ceph)
debug 2025-01-16T15:28:17.391+0000 7fba6e85a840 0 ceph version 19.2.0
(16063ff2022298c9300e49a547a16ffda59baf13) squid (stable), process radosgw, pid
7
debug 2025-01-16T15:28:17.391+0000 7fba6e85a840 0 framework: beast
debug 2025-01-16T15:28:17.391+0000 7fba6e85a840 0 framework conf key:
ssl_port, val: 7480
debug 2025-01-16T15:28:17.391+0000 7fba6e85a840 0 framework conf key:
ssl_certificate, val: /etc/ssl/certs/server.crt
debug 2025-01-16T15:28:17.391+0000 7fba6e85a840 0 framework conf key:
ssl_private_key, val: /etc/ssl/private/server.key
debug 2025-01-16T15:28:17.767+0000 7fba6e85a840 -1 LDAP not started since no
server URIs were provided in the configuration.
debug 2025-01-16T15:28:17.781+0000 7fba6e85a840 0 framework: beast
debug 2025-01-16T15:28:17.781+0000 7fba6e85a840 0 framework conf key:
ssl_certificate, val: config://rgw/cert/$realm/$zone.crt
debug 2025-01-16T15:28:17.781+0000 7fba6e85a840 0 framework conf key:
ssl_private_key, val: config://rgw/cert/$realm/$zone.key
debug 2025-01-16T15:28:17.860+0000 7fba6e85a840 0 starting handler: beast
debug 2025-01-16T15:28:17.908+0000 7fba6e85a840 0 set uid:gid to 167:167
(ceph:ceph)
debug 2025-01-16T15:28:17.912+0000 7fba6e85a840 1 mgrc service_daemon_register
rgw.643295 metadata {arch=x86_64,ceph_release=squid,ceph_version=ceph version
19.2.0 (16063ff2022298c9300e49a547a16ffda59baf13) squid
(stable),ceph_version_short=19.2.0,container_hostname=raynor-sc-1,container_image=ceph/squid:v19.2.0,cpu=Intel(R)
Xeon(R) CPU E5-2683 v3 @ 2.00GHz,distro=centos,distro_description=CentOS
Stream 9,distro_version=9,frontend_config#0=beast ssl_port=7480
ssl_certificate=/etc/ssl/certs/server.crt
ssl_private_key=/etc/ssl/private/server.key,frontend_type#0=beast,hostname=raynor-sc-1,id=25069123.raynor-sc-1.qiatgr,kernel_description=#1
SMP Thu Dec 19 20:58:14 EST
2024,kernel_version=5.4.288-1.el8.elrepo.x86_64,mem_swap_kb=0,mem_total_kb=12251748,num_handles=1,os=Linux,pid=7,realm_id=c99973d7-27cb-4abd-8391-bfbe4fa928be,realm_name=geored_realm,zone_id=30816d1d-e315-471b-bc84-6cdc4f6f1408,zone_name=siteA,zonegroup_id=c109a213-6638-4028-b21b-8bc9fcb49cd3,zonegroup_name=geored_zg}
Specifying the certificate by file path I think is the right choice for me as:
*
I don't have a domain, I have a per host certificate covering the IP of the
host. Each is different, signed by a self-signed trusted CA.
*
I don't want to put the hardcoded key into the service spec, as suggested in
the docs, as it appears in the mgr logs. My org policy (sensibly) doesn't allow
for secrets to appear in logs.
*
It's just generally more hassle to manage keys on a per VM basis, rather than a
static file path.
*
It will also allow me to not manage RGW services on a per host basis, but use
the same spec for all and a count=1 placement, which would be lovely.
So my question - is there a way around this such that I can specify the key and
certificate by file path and not have to manually change the config to make it
work?
Thanks,
Alex
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
