[ceph-users] Multisite RGW with Self-signed CA & Disconnected Upgrade

Hi ceph-users! I'm going through the process of migrating to use cephadm for my clusters. Previously I used ceph-ansible. I have a few questions related to this. 1. How can I configure RGW multisite with self-signed certificates? I have prototyped the migration and redeployed RGWs. Everythin

[ceph-users] RGW Multisite with a Self-Signed CA

Hi ceph-users! I'm going through the process of migrating to use cephadm for my clusters. Previously I used ceph-ansible. My question is essentially "How can I configure RGW multisite with self-signed certificates with cephadm?". I have prototyped the migration and redeployed RGWs. Everything o

[ceph-users] Re: [EXTERNAL] Re: RGW Multisite with a Self-Signed CA

x27;ll add that as well during a maintenance window, but for now it's not required. Regards, Eugen Zitat von "Alex Hussein-Kershaw (HE/HIM)" : > Hi ceph-users! > > I'm going through the process of migrating to use cephadm for my > clusters. Previously I used ceph-a

[ceph-users] Re: [EXTERNAL] Re: RGW Multisite with a Self-Signed CA

osgw-admin container and running "radosgw-admin sync status" shows looks much happier. I presume I could also mount the /etc/pki directory into my cephadm shell container if I desired also. Best wishes, Alex ________ From: Alex Hussein-Kershaw (HE/HIM) Se

[ceph-users] Cephadm Offline Bootstrapping Issue

Hi, I'm hitting an issue doing an offline install of Ceph 18.2.2 using cephadm. Long output below... any advice is appreciated. Looks like we don't managed to add admin labels (but also trying with --skip-admin results in a similar health warning). Subsequently trying to add an OSD fails quiet

[ceph-users] Re: [EXTERNAL] Re: Cephadm Offline Bootstrapping Issue

t. I'll explore that a bit. Thanks, Alex From: Adam King Sent: Friday, August 2, 2024 4:21 PM To: Alex Hussein-Kershaw (HE/HIM) Cc: ceph-users@ceph.io Subject: [EXTERNAL] Re: [ceph-users] Cephadm Offline Bootstrapping Issue The thing that stands out to m

[ceph-users] Re: [EXTERNAL] RGW bucket notifications stop working after a while and blocking requests

Hi Florian, We are also gearing up to use persistent bucket notifications, but have not got as far as you yet so quite interested in this. As I understand it, a bunch of new function is coming in Squid on the radosgw-admin command to allow gathering metrics from the queues, but they are not ava

[ceph-users] Cephadm and the "--data-dir" Argument

Hi Folks, I'm trying to use the --data-dir argument of cephadm when bootstrapping a Storage Cluster. It looks like exactly what I need, where my use case is that I want to data files onto a persistent disk, such that I can below away my VMs while retaining the files. Everything looks good and

[ceph-users] Re: [EXTERNAL] Re: Cephadm and the "--data-dir" Argument

ook at implementing a fix. Best Wishes, Alex From: Adam King Sent: Monday, August 12, 2024 4:05 PM To: Alex Hussein-Kershaw (HE/HIM) Cc: ceph-users ; Joseph Silva Subject: [EXTERNAL] Re: [ceph-users] Cephadm and the "--data-dir" Argument Looking through

[ceph-users] Re: [EXTERNAL] Re: Cephadm and the "--data-dir" Argument

, August 12, 2024 4:48 PM To: Alex Hussein-Kershaw (HE/HIM) Cc: ceph-users ; Joseph Silva Subject: Re: [EXTERNAL] Re: [ceph-users] Cephadm and the "--data-dir" Argument I think if it was locked in from bootstrap time it might not be that complicated. We'd just have to store the dir

[ceph-users] Cephadm Upgrade Issue

Hi Folks, I'm prototyping the upgrade process for our Ceph Clusters. I've adopted the Cluster following the docs, that works nicely 🙂 I then load my docker image into a locally running container registry, as I'm in a disconnected environment. I have a test Cluster with 3 VMs and no data, adopt

[ceph-users] Re: Cephadm Upgrade Issue

From: Alex Hussein-Kershaw (HE/HIM) Sent: Wednesday, August 14, 2024 3:06 PM To: ceph-users Subject: Cephadm Upgrade Issue Hi Folks, I'm prototyping the upgrade process for our Ceph Clusters. I've adopted the Cluster following the docs, that works nicely 🙂 I then load my docker i

[ceph-users] Persistent Bucket Notification Reconfiguring with CreateTopic

Hi folks, Currently running a test Ceph cluster on 19.1.0 and seeing some odd behaviour using the CreateTopic API. Initially I call the CreateTopic API, and all looks well: $ rga topic list { "topics": [ { "topics": [ { "owner": "ahk2"

[ceph-users] Re: Persistent Bucket Notification Reconfiguring with CreateTopic

I raised Bug #67765: Updating a Topic via the CreateTopic API Corrupts the Topic - rgw - Ceph<https://tracker.ceph.com/issues/67765> for this issue. Kind regards, Alex From: Alex Hussein-Kershaw (HE/HIM) Sent: Tuesday, August 27, 2024 6:01 PM To: ceph-use

[ceph-users] Bucket Notifications v2 & Multisite Redundancy

Hi Folks, I see in the pending release notes for Squid a description of the "notification_v2" feature. ceph/PendingReleaseNotes at main · ceph/ceph (github.com) I have some concerns about the multisite nature of this feature, I am

[ceph-users] Re: [EXTERNAL] Re: Bucket Notifications v2 & Multisite Redundancy

which is what I get with the pubsub module, and I think is what I will get with "notifications_v1". Best Wishes, Alex From: Yuval Lifshitz Sent: Tuesday, September 3, 2024 12:15 PM To: Alex Hussein-Kershaw (HE/HIM) Cc: ceph-users Subject: [EXT

[ceph-users] Re: [EXTERNAL] Re: Prefered distro for Ceph

Have been running on Centos/RHEL for ~5 years and not had any issues either. From: Roberto Maggi @ Debian Sent: Thursday, September 5, 2024 12:58 PM To: ceph-users@ceph.io Subject: [EXTERNAL] [ceph-users] Re: Prefered distro for Ceph Hi, I never tried anything e

[ceph-users] Re: [EXTERNAL] Re: Bucket Notifications v2 & Multisite Redundancy

gards, Alex From: Yuval Lifshitz Sent: Tuesday, September 3, 2024 2:56 PM To: Alex Hussein-Kershaw (HE/HIM) Cc: ceph-users Subject: Re: [EXTERNAL] Re: [ceph-users] Bucket Notifications v2 & Multisite Redundancy responded inline On Tue, Sep 3, 2024

[ceph-users] Re: [EXTERNAL] Re: Bucket Notifications v2 & Multisite Redundancy

uot;topics": [ { "topics": [ ..., ... ], } ] } Thanks, Alex From: Yuval Lifshitz Sent: Monday, September 16, 2024 1:25 PM To: Alex Hussein-Kershaw (HE/HIM) Cc: ceph-users Subject: Re: [EXTERNAL] Re: [ceph-users] Bucket Notifications v

[ceph-users] Re: [EXTERNAL] Re: Bucket Notifications v2 & Multisite Redundancy

ot; }, "arn": "arn:aws:sns:geored_zg::ahk2", "opaqueData": "", "policy": "" } ] } ] } So maybe a bug in Squid? __

[ceph-users] Is a direct Octopus to Reef Upgrade Possible?

Hi ceph-users, I currently use Ceph Octopus to provide CephFS & S3 Storage for our app servers, deployed in containers by ceph-ansible. I'm planning to take an upgrade to get off Ceph Octopus as it's EOL. I'd love to go straight to reef, but vaguely remember reading a statement that only two m

[ceph-users] Re: Multisite Config / Period Revert

uot;forget the period" -- but fixing the config seemed to do the job. -Original Message----- From: Alex Hussein-Kershaw (HE/HIM) Sent: Friday, September 16, 2022 5:37 PM To: Ceph Users Subject: [EXTERNAL] [ceph-users] Multisite Config / Period Revert Hi, I have a multisite system

[ceph-users] Large OMAP Objects & Pubsub

Hi All, Looking to get some advice on an issue my clusters have been suffering from. Realize there are lots of text below. Thanks in advance for your consideration. The cluster has a health warning of "32 large omap objects". It's persisted for several months. It appears functional and there a

[ceph-users] Re: [EXTERNAL] Re: Fw: Large OMAP Objects & Pubsub

s no multisite sync. So my question to any who may be able to answer: * Is the above analysis sound? * Can I update the zone config and delete these data_log objects manual to restore my cluster to HEALTH_OK? Thanks, Alex [1] https://access.redhat.com/solutions/6450561 [2] https://www.spin

[ceph-users] S3 Deletes in Multisite Sometimes Not Syncing

Hi Folks, Have made a strange observation on one of our Storage Clusters. * Running Ceph 15.2.13. * Set up as a multisite pair of siteA and siteB. The two sites are geographically separated. * We are using S3 with a bucket in versioning suspended state (we previously had versioning

[ceph-users] Re: [EXTERNAL] Re: S3 Deletes in Multisite Sometimes Not Syncing

hear from anyone who has replication working perfectly. On 2022-12-22 07:17, Alex Hussein-Kershaw (HE/HIM) wrote: > Hi Folks, > > Have made a strange observation on one of our Storage Clusters. > >* Running Ceph 15.2.13. >* Set up as a multisite pair of siteA and site

[ceph-users] How to Speed Up Draining OSDs?

Hi Folks, I'm trying to scale-in a Ceph Cluster. It's running 19.2.0 and is cephadm managed. It's just a test system, so has basically no data and only has 3 OSDs. As part of the scaling-in, I run "ceph orch host drain --zap-osd-devices" as per Host Management — Ceph Documentation

[ceph-users] Re: [EXTERNAL] Re: How to Speed Up Draining OSDs?

My pool size is indeed 3. Operator error 🙂 Thanks again, Alex From: Eugen Block Sent: Monday, October 21, 2024 3:08 PM To: Alex Hussein-Kershaw (HE/HIM) Cc: ceph-users@ceph.io Subject: Re: [EXTERNAL] [ceph-users] Re: How to Speed Up Draining OSDs? If your

[ceph-users] Re: [EXTERNAL] Re: How to Speed Up Draining OSDs?

tly know too little about mclock to know the right knobs. So far I've only tried it with only a few different settings and none helped significantly. I would expect that there are existing tracker issues since this topic comes up every other week or so. If not, I'd suggest to create

[ceph-users] Ceph Multisite Version Compatibility

Hi folks. I'm looking for some guidance on RGW multisite version sync compatibility. Particularly between Octopus and Squid. Context is I have two sites in a multisite pair replicating all S3 data. One is on Squid, one is on Octopus. Should I expect the multisite sync to just work between these

[ceph-users] Re: Ceph Multisite Version Compatibility

objects to SiteB, they appear on siteA shortly after. * Write objects to SiteA, they never appear on siteB. It seems to be 100% reproducible. Suspect I need to raise a tracker. I welcome any suggestions that I'm doing this wrong meanwhile. ____________________ From: Alex Hussein-Ke

[ceph-users] Re: [EXTERNAL] Multisite sync: is metadata transferred in plain text?

Feels like you answered your own question here - why not just use HTTPS for your multisite sync? I'm not aware of any other encryption mechanisms for metadata (I guess using SSE encryption for your data may save that going over in plain text) and would assume that just using HTTP would indeed b

[ceph-users] Re: [EXTERNAL] Re: Backup strategies for rgw s3

We have been using Amazon S3 (rclone.org) to copy all the data to a filesystem nightly to provide an S3 backup mechanism. It has Ceph support out the box (added by one of my colleagues a few years ago). From: Adam Prycki Sent: Wednes

[ceph-users] Re: [EXTERNAL] Re: Bucket Notifications v2 & Multisite Redundancy

tion configuration - rgw - Ceph<https://tracker.ceph.com/issues/68104> Also re-including the mailing list as it was dropped. From: Yuval Lifshitz Sent: Tuesday, September 17, 2024 10:36 AM To: Alex Hussein-Kershaw (HE/HIM) Subject: Re: [EXTERNAL] Re:

[ceph-users] Re: [EXTERNAL] Deploy rgw different version using cephadm

I think the advice is not to use floating tags (i.e. "latest") and use specific tags if possible. I believe you can achieve what you want with either: "ceph orch upgrade --image " not sure if this allows you to downgrade, but certainly lets you upgrade and change image, see Upgrading Ceph — Cep

[ceph-users] Re: [EXTERNAL] Re: Bucket Notifications v2 & Multisite Redundancy

he inability of notification config to remain single site on the latest Squid RC. Given the conversation we had below I think this is a bug. Happy to raise a tracker. Welcome any thoughts. I'll try to repro this on Reef shortly. Thanks, Alex From: Alex Hussein-

[ceph-users] Re: [EXTERNAL] Re: Bucket Notifications v2 & Multisite Redundancy

multisite even with notifications_v2 disabled - rgw - Ceph<https://tracker.ceph.com/issues/68227?next_issue_id=68211>. From: Alex Hussein-Kershaw (HE/HIM) Sent: Wednesday, September 25, 2024 10:42 AM To: Yuval Lifshitz ; Ceph Users Subject: Re: [EXTERNAL] Re:

[ceph-users] Re: [EXTERNAL] Re: Ceph Multisite Version Compatibility

dn't expect one sync direction to fail entirely. Maybe it's a configuration issue? Has it ever worked before or have you just set up the second site and it failed right from the start? Zitat von "Alex Hussein-Kershaw (HE/HIM)" : > I wondered if this applies: Ceph Releases

[ceph-users] Re: [EXTERNAL] Re: Ceph Multisite Version Compatibility

hat: * full sync is working * incremental sync is not ____________ From: Alex Hussein-Kershaw (HE/HIM) Sent: Wednesday, November 6, 2024 3:27 PM To: Eugen Block ; ceph-users@ceph.io Subject: Re: [EXTERNAL] [ceph-users] Re: Ceph Multisite Version Compatibility Hi Eugen, Than

[ceph-users] ceph orch ls --refresh

Hi Folks, Curious about the --refresh argument on this command (and similar). From experience so far, and a bit of a code read, it seems that: * Passing --refresh will kick the serve loop to run. * It won't wait for it to complete, so data returned may not be the latest. * Re-running the "

[ceph-users] Re: [EXTERNAL] Re: Cephadm: Specifying RGW Certs & Keys By Filepath

Oh actually I have spoke to soon. That does work, but it also exposes port HTTP over port 80. 🙁   beast port=80 ssl_port=7480 ssl_certificate=/etc/ssl/certs/server.crt ssl_private_key=/etc/ssl/private/server.key From: Alex Hussein-Kershaw (HE/HIM) Sent

[ceph-users] Re: Cephadm: Specifying RGW Certs & Keys By Filepath

.com/issues/69567>, and will submit a PR. ________ From: Alex Hussein-Kershaw (HE/HIM) Sent: Thursday, January 16, 2025 3:36 PM To: ceph-users Subject: Cephadm: Specifying RGW Certs & Keys By Filepath Hi Folks, Looking for some advice on RGW service specs and Cephadm.

[ceph-users] Re: [EXTERNAL] Re: Cephadm: Specifying RGW Certs & Keys By Filepath

Amazing. How did I miss that. Dropping "ssl: true" and adding "ssl_port=1234" to the rgw_frontend_extra_args values has me sorted. Many thanks! From: Redouane Kachach Sent: Thursday, January 16, 2025 4:39 PM To: Alex Hussein-Kershaw (HE

[ceph-users] Cephadm: Specifying RGW Certs & Keys By Filepath

Hi Folks, Looking for some advice on RGW service specs and Cephadm. I've read the docs here: RGW Service — Ceph Documentation Using a service spec I can deploy a RGW: service_type: rgw service_id: '25069123' service_name: rgw.25069123 placem

[ceph-users] Re: [EXTERNAL] Re: Cephadm: Specifying RGW Certs & Keys By Filepath

hanks, Alex From: Redouane Kachach Sent: Thursday, January 16, 2025 8:00 PM To: Alex Hussein-Kershaw (HE/HIM) Cc: ceph-users Subject: Re: [EXTERNAL] Re: [ceph-users] Cephadm: Specifying RGW Certs & Keys By Filepath That's strange... in the code I can see th

[ceph-users] Re: [EXTERNAL] Re: Cephadm: Specifying RGW Certs & Keys By Filepath

behaviour. Thanks, Alex From: Redouane Kachach Sent: Friday, January 17, 2025 11:14 AM To: Alex Hussein-Kershaw (HE/HIM) Cc: ceph-users Subject: Re: [EXTERNAL] Re: [ceph-users] Cephadm: Specifying RGW Certs & Keys By Filepath Thank you Alex, Reading the description of the tracker https://