[ceph-users] Re: Replacing OSD with containerized deployment

[mailing-lists]

Hey,

no problem and thank you!


This is the output of lsblk:


sda 8:0    0  14.6T  0 disk
└─ceph--937823b8--204b--4190--9bd1--f867e64621db-osd--block--a4bbaa5d--eb2d--41f3--8f4e--f8c5a2747012 
253:24   0  14.6T  0 lvm

sdb 8:16   0  14.6T  0 disk
└─ceph--169752b2--8095--41e9--87f2--cc6e962231ed-osd--block--8802416f--2e00--4388--847e--0e44b4a3afe2 
253:35   0  14.6T  0 lvm

sdc 8:32   0  14.6T  0 disk
└─ceph--8373a100--9457--43e9--a7c4--d573c2be9c0c-osd--block--7f582b9b--1433--4151--a644--6528e6991f75 
253:25   0  14.6T  0 lvm

sdd 8:48   0  14.6T  0 disk
└─ceph--5f830838--dbcf--4071--9886--5fff17a44590-osd--block--d9b18b2c--0bd3--4a9f--a22c--7a5b27abda3c 
253:30   0  14.6T  0 lvm

sde 8:64   0  14.6T  0 disk
└─ceph--aea40ec8--547e--4ebc--86af--b773a45ac857-osd--block--6dce53bc--1aef--4ee5--95c0--b2b9bcdb5733 
253:19   0  14.6T  0 lvm

sdf 8:80   0  14.6T  0 disk
└─ceph--0a6669a8--7c6f--4fef--9b49--46b0a6a62ece-osd--block--859896ad--e839--42d6--a84c--da244454bc6d 
253:34   0  14.6T  0 lvm

sdg 8:96   0  14.6T  0 disk
└─ceph--cf96ae54--eb04--4ef1--a162--6c647e23a139-osd--block--65a1f8a8--bdaa--4ffd--9c1b--2d60381a33fc 
253:29   0  14.6T  0 lvm

sdh 8:112  0  14.6T  0 disk
└─ceph--67deea47--0236--4392--b0c1--ecd5b30be3c6-osd--block--a7a03d63--91a6--4207--8192--ab078d4d596b 
253:22   0  14.6T  0 lvm

sdi 8:128  0  14.6T  0 disk
└─ceph--eef96e23--7f9e--458e--bc54--028c1161e11a-osd--block--10c6bdc6--9071--4c2f--b29e--7e5320a19f01 
253:20   0  14.6T  0 lvm

sdj 8:144  0  14.6T  0 disk
└─ceph--979b74d7--ac00--4e40--8e20--c45c274d8c3e-osd--block--eecdd1d8--1b2e--4e96--9914--b91623932bae 
253:33   0  14.6T  0 lvm

sdk 8:160  0  14.6T  0 disk
└─ceph--c94872dc--2567--4d73--b12c--0ab5bf700889-osd--block--72a52d58--06df--423f--89d6--a20b92f784b7 
253:32   0  14.6T  0 lvm

sdl 8:176  0  14.6T  0 disk
└─ceph--e480c934--f576--4a4c--821a--8328e2b23137-osd--block--9d67203d--37bb--4f36--9153--81550e1389db 
253:28   0  14.6T  0 lvm

sdm 8:192  0  14.6T  0 disk
└─ceph--1a529e9a--b15c--4b04--afe9--64f18a728c63-osd--block--70abcdcf--88c2--4085--b5e0--2f028ad946a1 
253:36   0  14.6T  0 lvm

sdn 8:208  0  14.6T  0 disk
└─ceph--3e15ae4d--72f3--4c8d--979d--e00fabc5fe99-osd--block--ff7ebdca--afe0--477c--81a7--0773b6449487 
253:26   0  14.6T  0 lvm

sdo 8:224  0  14.6T  0 disk
└─ceph--81805753--ae18--4416--87f7--3a996ece90f3-osd--block--4ae6d812--2e33--4fbd--8d79--f2260af1765f 
253:23   0  14.6T  0 lvm

sdp 8:240  0  14.6T  0 disk
└─ceph--6776033b--3a45--47fe--aa5e--45e82602a10d-osd--block--67f8fe64--8cc7--43e1--857a--8d59fd5d0f81 
253:31   0  14.6T  0 lvm

sdq 65:0    0  14.6T  0 disk
└─ceph--a3ee82aa--0811--4a4d--80fc--ada3ba93376f-osd--block--f5d90973--eed3--4a1b--923d--a39263cb8546 
253:21   0  14.6T  0 lvm

sdr 65:16   0  14.6T  0 disk
└─ceph--29846433--9bf8--4c2c--89b8--054af1f301c7-osd--block--f2910199--3df5--488e--aa2a--4ccd98ba6453 
253:27   0  14.6T  0 lvm

sds 65:32   0   400G  0 disk
├─sds1 65:33   0 1G  0 part /boot/efi
├─sds2 65:34   0 2G  0 part /boot
└─sds3 65:35   0 396.9G  0 part
└─ubuntu--vg-ubuntu--lv 253:18   0   300G  0 lvm  /
nvme1n1 259:2    0   5.8T  0 disk
├─ceph--b38117e8--8e50--48dd--95f2--b4226286bfde-osd--wal--09370408--a1a5--4d32--9a15--9da8ccac931d 
253:0    0 331.2G  0 lvm
├─ceph--b38117e8--8e50--48dd--95f2--b4226286bfde-osd--wal--324bb7f9--afac--49d3--910a--73643f6c09b2 
253:1    0 331.2G  0 lvm
├─ceph--b38117e8--8e50--48dd--95f2--b4226286bfde-osd--wal--e65bf85c--96d2--4ae2--b9d1--9a60f957ab97 
253:2    0 331.2G  0 lvm
├─ceph--b38117e8--8e50--48dd--95f2--b4226286bfde-osd--wal--a7f4cf98--8e1a--4d8a--b622--787d5db4ee3e 
253:3    0 331.2G  0 lvm
├─ceph--b38117e8--8e50--48dd--95f2--b4226286bfde-osd--wal--bb925b1b--6d02--4be6--aa4f--254e4f1e8202 
253:4    0 331.2G  0 lvm
├─ceph--b38117e8--8e50--48dd--95f2--b4226286bfde-osd--wal--fff0b13d--a90c--4ada--a6fa--20e98cb956c8 
253:5    0 331.2G  0 lvm
├─ceph--b38117e8--8e50--48dd--95f2--b4226286bfde-osd--wal--139637be--7bc9--4398--8461--3e88382a9eaa 
253:6    0 331.2G  0 lvm
├─ceph--b38117e8--8e50--48dd--95f2--b4226286bfde-osd--wal--a274b742--65ba--4bf3--918e--166023565c69 
253:7    0 331.2G  0 lvm
└─ceph--b38117e8--8e50--48dd--95f2--b4226286bfde-osd--wal--d7efd132--5079--4ae5--aaa2--d57e05e86fb6 
253:8    0 331.2G  0 lvm

nvme0n1 259:3    0   5.8T  0 disk
├─ceph--3a336b8e--ed39--4532--a199--ac6a3730840b-osd--wal--50092d2d--f06a--47f9--adce--ca9344d5615f 
253:9    0 331.2G  0 lvm
├─ceph--3a336b8e--ed39--4532--a199--ac6a3730840b-osd--wal--cc1d8423--d699--4b69--a426--87f01d289e2d 
253:10   0 331.2G  0 lvm
├─ceph--3a336b8e--ed39--4532--a199--ac6a3730840b-osd--wal--f5b7018b--8537--4153--b6f5--2ecbe4d1f109 
253:11   0 331.2G  0 lvm
├─ceph--3a336b8e--ed39--4532--a199--ac6a3730840b-osd--wal--5d845dba--8b55--4984--890b--547fbdaff10c 
253:12   0 331.2G  0 lvm
├─ceph--3a336b8e--ed39--4532--a199--ac6a3730840b-osd--wal--e100d725--fa1e--47d5--844b--d7fdca8093ca 
253:13   0 331.2G  0 lvm
├─ceph--3a336b8e--ed39--4532--a199--ac6a3730840b-osd--wal--62f7a57b--dc82--45c9

[ceph-users] Adding osds to each nodes

[Szabo, Istvan (Agoda)]

Hi,

What is the safest way to add disk(s) to each of the node in the cluster?
Should it be done 1 by 1 or can add all of them at once and let it rebalance?

My concern is that if add all in one due to host based EC code it will block 
all the host.
The other side if I add 1 by 1,  one node will have more performance and more 
osds than the others which is also not a good setup, so wonder which is the 
safer way?

(have 9 nodes with host based EC 4:2, 1 disk is going to have 4osds)

Thank you


This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: RGW archive zone lifecycle

[Matt Benjamin]

Hi Ondřej,

Yes, we added an extension to allow writing lifecycle policy which will
only take effect in archive zone(s).  It's currently present on ceph/main,
and will be in Reef.

Matt

On Wed, Feb 8, 2023 at 2:10 AM Ondřej Kukla  wrote:

> Hi,
>
> I have two Ceph clusters in a multi-zone setup. The first one (master
> zone) would be accessible to users for their interaction using RGW.
> The second one is set to sync from the master zone with the tier type of
> the zone set as an archive (to version all files).
>
> My question here is. Is there an option to set a lifecycle for the version
> files saved on the archive zone? For example, keep only 5 versions per file
> or delete version files older than one year?
>
> Thanks a lot.
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
>

-- 

Matt Benjamin
Red Hat, Inc.
315 West Huron Street, Suite 140A
Ann Arbor, Michigan 48103

http://www.redhat.com/en/technologies/storage

tel.  734-821-5101
fax.  734-769-8938
cel.  734-216-5309
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] OSD logs missing from Centralised Logging

[Peter van Heusden]

Hi there

I am running Ceph version 17.2.5 and have deployed centralised logging as
per this guide:

https://ceph.io/en/news/blog/2022/centralized_logging/

The logs from the OSDs are not, however, showing up in the Grafana
dashboard, as per this screenshot:

[image: image.png]
The Promtail daemons are running on each node including the OSDs. The Loki
server and Grafana are running on one of our monitor nodes.

Thanks for any clarifications you can provide.

Peter
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Nautilus to Octopus when RGW already on Octopus

[Eugen Block]

Hi,

I also would try to avoid a downgrade and agree with Richard. From a  
ceph perspective the RGWs are just clients and not core services so I  
wouldn't worry too much if they already are on a newer version.  
Although I didn't test this specific scenario either there's one  
cluster we help maintain that is comparable to this, ceph is still  
running Nautilus but the CephFS kernel clients are running on Octopus  
already. We don't see any issues with that.


Regards,
Eugen


Zitat von Richard Bade :


Hi,
We're actually on very similar setup to you with 18.04 and Nautilus
and thinking about the 20.04 upgrade process.

As for your RGW, I think I would not consider the downgrade. I believe
the order is about avoiding issues with newer RGW connecting to older
mons and osds. Since you're already in this situation and not having
any issues, I would probably continue forward with the upgrade on
Mons, then Managers, then osds as per documentation. Then just restart
the RGW at the end.
I think that trying to downgrade at this point may introduce new
issues that you don't currently have.

This is just my opinion though, as I have not actually tried this. Do
you have a test cluster you could practice on?
I would be keen to hear how your upgrade goes.

Regards,
Richard

On Sat, 4 Feb 2023 at 22:10,  wrote:


We are finally going to upgrade our Ceph from Nautilus to Octopus,  
before looking at moving onward.  We are still on Ubuntu 18.04, so  
once on Octopus, we will then upgrade the OS to 20.04, ready for  
the next upgrade.


Unfortunately, we have already upgraded our rados gateways to  
Ubuntu 20.04, last Sept, which had the side effect of upgrading the  
RGWs to Octopus. So I'm looking to downgrade the rados gateways,  
back to Nautilus, just to be safe.  We can then do the upgrade in  
the right order.


I have no idea if the newer Octopus rados gateways will have  
altered any metadata, that would affect a downgrade back to Nautilus.


Any advise.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io



___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: OSD upgrade problem nautilus->octopus - snap_mapper upgrade stuck

[Eugen Block]

Hi,

Someone told me that we could just destroy the FileStore OSD’s and  
recreate them as BlueStore, even though the cluster is partially  
upgraded. So I guess I’ll just do that. (Unless someone here tells  
me that that’s a terrible idea :))


I would agree, rebuilding seems a reasonable approach here. The SUSE  
documentation [1] doesn't even allow Filestore OSDs before an upgrade  
to a cephadm managed cluster. I don't have tried to upgrade Filestore  
based clusters so I can't tell if this is related to Filestore in  
general or a different issue. Anyway, moving towards Bluestore is  
inevitable, so at some point you'll need to rebuild anyway. :-)


Regards,
Eugen

[1]  
https://documentation.suse.com/ses/7.1/single-html/ses-deployment/#before-upgrade



Zitat von Mark Schouten :


Hi,

Thanks. Someone told me that we could just destroy the FileStore  
OSD’s and recreate them as BlueStore, even though the cluster is  
partially upgraded. So I guess I’ll just do that. (Unless someone  
here tells me that that’s a terrible idea :))


—
Mark Schouten, CTO
Tuxis B.V.
m...@tuxis.nl / +31 318 200208


-- Original Message --
From "Eugen Block" 
To ceph-users@ceph.io
Date 2/7/2023 4:58:11 PM
Subject [ceph-users] Re: OSD upgrade problem nautilus->octopus -  
snap_mapper upgrade stuck



Hi,

I don't really have an answer, but there was a bug with snap mapper  
 [1], [2] is supposed to verify consistency, but Octopus is EOL so  
you  might need to upgrade directly to Pacific. That's what we did  
on  multiple clusters (N --> P) a few months back. I'm not sure if  
it  would just work if you already have a couple of Octopus  
daemons, maybe  you can try it on a test cluster.


Regards,
Eugen

[1] https://tracker.ceph.com/issues/56147
[2] https://github.com/ceph/ceph/pull/47388

Zitat von Mark Schouten :


Hi,

I’m seeing the same thing …

With debug logging enabled I see this:
2023-02-07T00:35:51.853+0100 7fdab9930e00 10   
snap_mapper.convert_legacy converted 1410 keys
2023-02-07T00:35:51.853+0100 7fdab9930e00 10   
snap_mapper.convert_legacy converted 1440 keys
2023-02-07T00:35:51.853+0100 7fdab9930e00 10   
snap_mapper.convert_legacy converted 1470 keys
2023-02-07T00:35:51.853+0100 7fdab9930e00 10   
snap_mapper.convert_legacy converted 1500 keys


It ends at 1500 keys. And nothing happens.

I’m now stuck with a cluster that has 4 OSD’s on Octopus, 10 on   
Nautilus, and one down .. A hint on how to work around this is   
welcome :)


—
Mark Schouten, CTO
Tuxis B.V.
m...@tuxis.nl / +31 318 200208


-- Original Message --
From "Jan Pekař - Imatic" 
To ceph-users@ceph.io
Date 1/12/2023 5:53:02 PM
Subject [ceph-users] OSD upgrade problem nautilus->octopus -   
snap_mapper upgrade stuck



Hi all,

I have problem upgrading nautilus to octopus on my OSD.

Upgrade mon and mgr was OK and first OSD stuck on

2023-01-12T09:25:54.122+0100 7f49ff3eae00  1 osd.0 126556 init   
upgrade snap_mapper (first start as octopus)


and there were no activity after that for more than 48 hours. No   
disk activity.


I restarted OSD many times and nothing changed.

It is old, filestore OSD based on XFS filesystem. Is upgrade to   
snap mapper 2 reliable? What is OSD waiting for? Can I start OSD   
without upgrade and get cluster healthy with old snap structure?  
Or  should I skip octopus upgrade and go to pacific directly  
(some bug  backport is missing?).


Thank you for help, I'm sending some logs below..

Log shows

2023-01-09T19:12:49.471+0100 7f41f60f1e00  0 ceph version 15.2.17  
 (694d03a6f6c6e9f814446223549caf9a9f60dba0) octopus (stable),   
process ceph-osd, pid 2566563
2023-01-09T19:12:49.471+0100 7f41f60f1e00  0 pidfile_write:  
ignore  empty --pid-file
2023-01-09T19:12:49.499+0100 7f41f60f1e00 -1 missing 'type' file,  
 inferring filestore from current/ dir
2023-01-09T19:12:49.531+0100 7f41f60f1e00  0 starting osd.0   
osd_data /var/lib/ceph/osd/ceph-0 /var/lib/ceph/osd/ceph-0/journal
2023-01-09T19:12:49.531+0100 7f41f60f1e00 -1 Falling back to  
public  interface
2023-01-09T19:12:49.871+0100 7f41f60f1e00  0 load: jerasure load:  
 lrc load: isa
2023-01-09T19:12:49.875+0100 7f41f60f1e00  0   
filestore(/var/lib/ceph/osd/ceph-0) backend xfs (magic 0x58465342)
2023-01-09T19:12:49.883+0100 7f41f60f1e00  0 osd.0:0.OSDShard  
using  op scheduler  
ClassedOpQueueScheduler(queue=WeightedPriorityQueue,  cutoff=196)
2023-01-09T19:12:49.883+0100 7f41f60f1e00  0 osd.0:1.OSDShard  
using  op scheduler  
ClassedOpQueueScheduler(queue=WeightedPriorityQueue,  cutoff=196)
2023-01-09T19:12:49.883+0100 7f41f60f1e00  0 osd.0:2.OSDShard  
using  op scheduler  
ClassedOpQueueScheduler(queue=WeightedPriorityQueue,  cutoff=196)
2023-01-09T19:12:49.883+0100 7f41f60f1e00  0 osd.0:3.OSDShard  
using  op scheduler  
ClassedOpQueueScheduler(queue=WeightedPriorityQueue,  cutoff=196)
2023-01-09T19:12:49.883+0100 7f41f60f1e00  0 osd.0:4.OSDShard  
using  op scheduler  
ClassedOpQueueScheduler(queue=WeightedPriorityQueue,  cutoff=1

[ceph-users] Re: Adding osds to each nodes

[Eugen Block]

Hi,

this is a quite common question and multiple threads exist on this  
topic, e.g. [1].


Regards,
Eugen

[1] https://www.mail-archive.com/ceph-users@lists.ceph.com/msg36475.html


Zitat von "Szabo, Istvan (Agoda)" :


Hi,

What is the safest way to add disk(s) to each of the node in the cluster?
Should it be done 1 by 1 or can add all of them at once and let it rebalance?

My concern is that if add all in one due to host based EC code it  
will block all the host.
The other side if I add 1 by 1,  one node will have more performance  
and more osds than the others which is also not a good setup, so  
wonder which is the safer way?


(have 9 nodes with host based EC 4:2, 1 disk is going to have 4osds)

Thank you


This message is confidential and is for the sole use of the intended  
recipient(s). It may also be privileged or otherwise protected by  
copyright or other legal rules. If you have received it by mistake  
please let us know by reply email and delete it from your system. It  
is prohibited to copy this message or disclose its content to  
anyone. Any confidentiality or privilege is not waived or lost by  
any mistaken delivery or unauthorized disclosure of the message. All  
messages sent to and from Agoda may be monitored to ensure  
compliance with company policies, to protect the company's interests  
and to remove potential malware. Electronic messages may be  
intercepted, amended, lost or deleted, or contain viruses.

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io



___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Exit yolo mode by increasing size/min_size does not (really) work

[Eugen Block]

Hi,

I don't have an explanation yet but some more information about your  
cluster would be useful like 'ceph osd tree', 'ceph osd df', 'ceph  
status' etc.


Thanks,
Eugen

Zitat von Stefan Pinter :


Hi! 😊

It would be very kind of you to help us with that!

We have pools in our ceph cluster that are set to replicated size 2  
min_size 1.
Obviously we want to go to size 3 / min_size 2 but we experience  
problems with that.


USED goes to 100% instantly and MAX AVAIL goes to 0. Write  
operations seemed to stop.


   POOLS:
   NAME   ID USED 
   %USED  MAX AVAIL OBJECTS
   Pool1 24 35791G  35.04 
66339G 8927762
   Pool2 25 11610G   
14.8966339G 3004740
   Pool3  26 17557G  
100.00 0 2666972



Before the change it was like this:

   NAME   
 ID USED   %USED MAX AVAIL OBJECTS
   Pool1  24 35791G  
35.0466339G 8927762
   Pool2 25 11610G 14.89  
   66339G 3004740
   Pool3  26 17558G  
20.9366339G 2667013



This was quite surprising to us as we’d expect USED to go to  
something like 30%.

Going back to 2/1 also gave us back the 20.93% usage instantly.

What’s the matter here?

Thank you and best regards
Stefan

BearingPoint GmbH
Sitz: Wien
Firmenbuchgericht: Handelsgericht Wien
Firmenbuchnummer: FN 175524z

The information in this email is confidential and may be legally  
privileged. If you are not the intended recipient of this message,  
any review, disclosure, copying, distribution, retention, or any  
action taken or omitted to be taken in reliance on it is prohibited  
and may be unlawful. If you are not the intended recipient, please  
reply to or forward a copy of this message to the sender and delete  
the message, any attachments, and any copies thereof from your system.

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io



___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Adding osds to each nodes

[Szabo, Istvan (Agoda)]

Ok, seems better to add all disks to host by hosts with waiting for rebalance 
between each of them, thx.

Istvan Szabo
Staff Infrastructure Engineer
---
Agoda Services Co., Ltd.
e: istvan.sz...@agoda.com
---

On 2023. Feb 8., at 20:43, Eugen Block  wrote:

Email received from the internet. If in doubt, don't click any link nor open 
any attachment !


Hi,

this is a quite common question and multiple threads exist on this
topic, e.g. [1].

Regards,
Eugen

[1] https://www.mail-archive.com/ceph-users@lists.ceph.com/msg36475.html


Zitat von "Szabo, Istvan (Agoda)" :

Hi,

What is the safest way to add disk(s) to each of the node in the cluster?
Should it be done 1 by 1 or can add all of them at once and let it rebalance?

My concern is that if add all in one due to host based EC code it
will block all the host.
The other side if I add 1 by 1,  one node will have more performance
and more osds than the others which is also not a good setup, so
wonder which is the safer way?

(have 9 nodes with host based EC 4:2, 1 disk is going to have 4osds)

Thank you


This message is confidential and is for the sole use of the intended
recipient(s). It may also be privileged or otherwise protected by
copyright or other legal rules. If you have received it by mistake
please let us know by reply email and delete it from your system. It
is prohibited to copy this message or disclose its content to
anyone. Any confidentiality or privilege is not waived or lost by
any mistaken delivery or unauthorized disclosure of the message. All
messages sent to and from Agoda may be monitored to ensure
compliance with company policies, to protect the company's interests
and to remove potential malware. Electronic messages may be
intercepted, amended, lost or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io


___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io


This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Is autoscaler doing the right thing?

[Kyriazis, George]

Hello ceph community,

I have some questions about the pg autoscaler.  I have a cluster with several 
pools.  One of them is a cephfs pool which is behaving in an expected / sane 
way, and another is a RBD pool with an ec profile of k=2, m=2.

The cluster has about 60 drives across across about 10 failure domains.  
(Failure domain is set to “chassis”, and there are some chassis with 4 blades 
per chassis, and the rest with 1 host per chassis).

The rbd ec pool has 66TiB stored with 128 PGs.  Each PG has about 500k objects 
in them, which seems like quite a lot.  When rebalancing, this EC pool is 
always the longpole.

The confusing part is that I am getting inconsistent output on the status on 
the autoscaler.  For example:

root@vis-mgmt:~# ceph osd pool autoscale-status | grep rbd_ec
rbd_ec67241G2.0856.7T  0.1533   
   1.0  64  on False  

Which tells me I have 64 PG_NUM (a lie).

root@vis-mgmt:~# ceph osd pool ls detail | grep rbd_ec
pool 4 'rbd_ec' erasure profile ec22 size 4 min_size 3 crush_rule 1 object_hash 
rjenkins pg_num 128 pgp_num 120 pg_num_target 64 pgp_num_target 64 
autoscale_mode on last_change 83396 lfor 0/83395/83393 flags 
hashpspool,ec_overwrites,selfmanaged_snaps stripe_width 8192 application rbd

Tells me I have 128 PGs (correct), but with a pgp_num which is not a power of 2 
(120 pgs).  Also, I am not sure what the pg_num_target and pgp_num_raget are 
and why they are different from pg_num and pgp_num.

Is there anything I can look into to find out is the autoscaler is working 
correctly for this pool?  Any other tweaks I need to do?  Seems to me that with 
that capacity it ought to have more than 128 PGs…

Thank you!

George

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io