[ceph-users] Impact of many objects per PG

[Eugen Block]

Hi *,

are there any known limitations or impacts of (too) many objects per  
PG? We're dealing with a performance decrease on Nautilus (I know, but  
it can't be upgraded at this time) while pushing a million emails  
(many small objects) into the cluster. At some point, maybe between  
600.000 and 900.000 emails or so, the client requests became slower  
(CephFS kernel clients) although all entities look fine, the MDS  
daemons are not overloaded, the OSDs not fully utilized (HDDs with  
shared rocksDB on SSDs). The HDD OSDs currently have around 100 PGs  
per OSD with 45 GB per PG (quite a lot) and around 180.000 objects per  
PG. The main data pool has an EC profile with k=4 m=5.
We want to increase the pg_num anyway, we expect a general performance  
increase after the pg splitting, but we're still wondering where the  
limits are. For example, if we increase mon_max_pg_per_osd to more  
than 250 we could split the PGs even more to reduce PG size and number  
of objects per PG, if the OSDs can cope with it utilization-wise, but  
I'm not sure if that's a good idea. Are there any other tunables we  
could tweak instead to reduce the performance impact?

Any comments or references are highly appreciated!

Thanks,
Eugen

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Impact of many objects per PG

[Eugen Block]

Thanks,

I found this thread [1] recommending offline compaction with large  
OMAP/META data on the OSDs. We'll try that first and see if it helps.  
We're still missing some details about the degredation, I'll update  
this thread when we know more.

The PGs are balanced quite perfectly and we don't have spillover.

[1]  
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/EDL7U5EWFHSFK5IIBRBNAIXX7IFWR5QK/



Zitat von "Szabo, Istvan (Agoda)" :


Hi,

Last year this caused my long lasting issue, the autoscaler  
suggestion was completely irrelevant about pg number.


I had an objectstore cluster with 2B objects, most of them small  
like 50kb, and the only solution to avoid the slow ops in my cluster  
was to:

- increase the pg number
- Use the balancer with max deviation 1
- Remove wal+rocksdb from a separated device because it was  
continuously spill overed

- 4osd for each 15TB ssds

Now the cluster is kind of stable in the last half year.

Istvan Szabo
Senior Infrastructure Engineer
---
Agoda Services Co., Ltd.
e: istvan.sz...@agoda.com
---

-Original Message-
From: Eugen Block 
Sent: Tuesday, July 26, 2022 4:39 PM
To: ceph-users@ceph.io
Subject: [ceph-users] Impact of many objects per PG

Email received from the internet. If in doubt, don't click any link  
nor open any attachment !



Hi *,

are there any known limitations or impacts of (too) many objects per  
PG? We're dealing with a performance decrease on Nautilus (I know,  
but it can't be upgraded at this time) while pushing a million  
emails (many small objects) into the cluster. At some point, maybe  
between
600.000 and 900.000 emails or so, the client requests became slower  
(CephFS kernel clients) although all entities look fine, the MDS  
daemons are not overloaded, the OSDs not fully utilized (HDDs with  
shared rocksDB on SSDs). The HDD OSDs currently have around 100 PGs  
per OSD with 45 GB per PG (quite a lot) and around 180.000 objects  
per PG. The main data pool has an EC profile with k=4 m=5.
We want to increase the pg_num anyway, we expect a general  
performance increase after the pg splitting, but we're still  
wondering where the limits are. For example, if we increase  
mon_max_pg_per_osd to more than 250 we could split the PGs even more  
to reduce PG size and number of objects per PG, if the OSDs can cope  
with it utilization-wise, but I'm not sure if that's a good idea.  
Are there any other tunables we could tweak instead to reduce the  
performance impact?

Any comments or references are highly appreciated!

Thanks,
Eugen

___
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an  
email to ceph-users-le...@ceph.io



This message is confidential and is for the sole use of the intended  
recipient(s). It may also be privileged or otherwise protected by  
copyright or other legal rules. If you have received it by mistake  
please let us know by reply email and delete it from your system. It  
is prohibited to copy this message or disclose its content to  
anyone. Any confidentiality or privilege is not waived or lost by  
any mistaken delivery or unauthorized disclosure of the message. All  
messages sent to and from Agoda may be monitored to ensure  
compliance with company policies, to protect the company's interests  
and to remove potential malware. Electronic messages may be  
intercepted, amended, lost or deleted, or contain viruses.




___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: librbd leaks memory on crushmap updates

[Peter Lieven]

Am 21.07.22 um 17:50 schrieb Ilya Dryomov:

On Thu, Jul 21, 2022 at 11:42 AM Peter Lieven  wrote:

Am 19.07.22 um 17:57 schrieb Ilya Dryomov:

On Tue, Jul 19, 2022 at 5:10 PM Peter Lieven  wrote:

Am 24.06.22 um 16:13 schrieb Peter Lieven:

Am 23.06.22 um 12:59 schrieb Ilya Dryomov:

On Thu, Jun 23, 2022 at 11:32 AM Peter Lieven  wrote:

Am 22.06.22 um 15:46 schrieb Josh Baergen:

Hey Peter,


I found relatively large allocations in the qemu smaps and checked the 
contents. It contained several hundred repetitions of osd and pool names. We 
use the default builds on Ubuntu 20.04. Is there a special memory allocator in 
place that might not clean up properly?

I'm sure you would have noticed this and mentioned it if it was so -
any chance the contents of these regions look like log messages of
some kind? I recently tracked down a high client memory usage that
looked like a leak that turned out to be a broken config option
resulting in higher in-memory log retention:
https://tracker.ceph.com/issues/56093. AFAICT it affects Nautilus+.

Hi Josh, hi Ilya,


it seems we were in fact facing 2 leaks with 14.x. Our long running VMs with 
librbd 14.x have several million items in the osdmap mempool.

In our testing environment with 15.x I see no unlimited increase in the osdmap 
mempool (compared this to a second dev host with 14.x client where I see the 
increase wiht my tests),

but I still see leaking memory when I generate a lot of osdmap changes, but 
this in fact seem to be log messages - thanks Josh.


So I would appreciate if #56093 would be backported to Octopus before its final 
release.

I picked up Josh's PR that was sitting there unnoticed but I'm not sure
it is the issue you are hitting.  I think Josh's change just resurrects
the behavior where clients stored only up to 500 log entries instead of
up to 1 (the default for daemons).  There is no memory leak there,
just a difference in how much memory is legitimately consumed.  The
usage is bounded either way.

However in your case, the usage is slowly but constantly growing.
In the original post you said that it was observed both on 14.2.22 and
15.2.16.  Are you saying that you are no longer seeing it in 15.x?

After I understood whats the background of Josh issue I can confirm that I 
still see increasing memory which is not caused

by osdmap items and also not by log entries. There must be something else going 
on.

I still see increased memory (heap) usage. Might it be that it is just heap 
fragmentation?

Hi Peter,

It could be but you never quantified the issue.  What is the actual
heap usage you are seeing, how fast is it growing?  Is it specific to
some particular VMs or does it affect the entire fleet?


Hi Ilya,


I see the issue across the fleet. The memory increases about 200KB/day per 
attached drive.

Same hypervisor with attached iSCSI storage - no issue.


However, the memory that is increasing is not listed as heap under 
/proc/{pid}/smaps.

Does librbd use its own memory allocator?

Hi Peter,

By default, librbd uses tcmalloc.



I am still testing with 15.x as I mainly have long running VMs in our 
production environment.

With 14.x we had an additional issue with the ospmaps not beeing freed. That is 
gone with 15.x


I will try with a patched qemu that allocated the write buffers inside qemu and 
set disable_zero_copy_write = true.

to see if this makes any difference.

We are unlikely to be able to do anything about 15.x at this point so
I'd encourage you to try 17.x.  That said, any new information would be
helpful.



Update from my side. Even with Quincy I see guest data in librbd memory even if 
disable_zero_copy_write is set to false.

How can that happen? Any ideas? rbd_cache is off.


Peter


___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: octopus v15.2.17 QE Validation status

[Josh Durgin]

On Sun, Jul 24, 2022 at 8:33 AM Yuri Weinstein  wrote:

> Still seeking approvals for:
>
> rados - Travis, Ernesto, Adam
> rgw - Casey
> fs, kcephfs, multimds - Venky, Patrick
> ceph-ansible - Brad pls take a look
>
> Josh, upgrade/client-upgrade-nautilus-octopus failed, do we need to fix
> it, pls take a look/approve.
>

Looks like a python2/3 issue, not worth the overhead at this point. Let's
proceed without this one.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] insecure global_id reclaim

[Dylan Griff]

Hello!

This is a bit of an older topic but were just hitting it now. Our cluster is 
still 14.2.22 (working on the upgrade) and we had the "mons are allowing 
insecure global_id reclaim" health warning. It took us awhile to update all our 
clients but after doing so I have set “auth_allow_insecure_global_id_reclaim” 
on the mons to false and all seems well. Warnings are gone and clients are 
still connecting, functionality seems good. However our logs on each OSD server 
are absolutely bursting with a few million lines a day like this:

user.info ceph-osd[5554]: 2022-07-26 00:00:00.821 7f67da91c700  0 cephx: 
verify_authorizer could not get service secret for service osd secret_id=34005
user.info ceph-osd[5570]: 2022-07-26 00:00:00.822 7f8df6121700  0 auth: could 
not find secret_id=34005

I have restarted the OSD processes with no change. I’m not sure how to see who 
or where is referencing these secret_ids that don’t seem to exist? Is there 
something I can do on the cluster side or the client side to try and forget 
them?

Thanks,
Dylan
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] large omap objects in the rgw.log pool

[Sarah Coxon]

Hi all,

We have 2 Ceph clusters in multisite configuration, both are working fine
(syncing correctly) but 1 of them is showing warning 32 large omap objects
in the log pool.

This seems to be coming from the sync error list

for i in `rados -p wilxite.rgw.log ls`; do echo -n "$i:"; rados -p
wilxite.rgw.log listomapkeys $i | wc -l; done > /tmp/omapkeys

sort -t: -k2 -r -n /tmp/omapkeys | head -1
sync.error-log.1:474102

This command 'radosgw-admin sync error list' shows a lot of very old
entries, none for 2022, how can I get rid of them? The trim command doesn't
do anything and both sites are showing data and metadata as up to date.

I have run a resync on the metadata and also data sync on a couple of
buckets mentioned in the error list but it's made no difference.

Would really appreciate any help as I have spent hours and hours trawling
through everything online and can't find any info on how I can clear the
error log.

Best Regards,
Sarah
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: LibCephFS Python Mount Failure

[Gregory Farnum]

It looks like you’re setting environment variables that force your new
keyring,  it you aren’t telling the library to use your new CephX user. So
it opens your new keyring and looks for the default (client.admin) user and
doesn’t get anything.
-Greg

On Tue, Jul 26, 2022 at 7:54 AM Adam Carrgilson (NBI) <
adam.carrgil...@nbi.ac.uk> wrote:

> I've disabled the part of the script that catches the Python exception and
> allowed it to print everything out and it looks like the OSError with the
> code 13, is a permissions error:
>
> Traceback (most recent call last):
>   File "./get-ceph-quota-statistics.py", line 274, in 
> main(args)
>   File "./get-ceph-quota-statistics.py", line 30, in main
> cephfs = login() # holds CephFS bindings
>   File "./get-ceph-quota-statistics.py", line 94, in login
> cephfs.mount(filesystem_name=b'cephfs')
>   File "cephfs.pyx", line 684, in cephfs.LibCephFS.mount
>   File "cephfs.pyx", line 676, in cephfs.LibCephFS.init
> cephfs.OSError: error calling ceph_init: Permission denied [Errno 13]
>
> Now I've tested a FUSE mount with the same keyfile and that functions as
> expected, so I'm having to assume that somehow the Python script either
> doesn't have all of the properties I've supplied (which I doubt, because I
> can point it at files with admin credentials and it works fine), something
> within the Python CephFS library might be hardcoded to particular values
> which I'm having problems with, or maybe something else?
>
> Is there a way to interrogate the Python object before I do the
> cephfs.mount, just to confirm the options are as I expect?
>
> Alternatively, python-cephfs wraps around the CephFS library, right?
> Does the CephFS FUSE component utilise the same CephFS library?
> If not, is there a way to call something else on the command line directly
> to rule out problems there?
>
> Many Thanks,
> Adam.
>
> -Original Message-
> From: Adam Carrgilson (NBI) 
> Sent: 25 July 2022 16:24
> To: ceph-users@ceph.io
> Cc: Bogdan Adrian Velica 
> Subject: [ceph-users] Re: LibCephFS Python Mount Failure
>
> Thanks Bogdan,
>
> I’m running this script at the moment as my development system’s root user
> account, I don’t have a particular ceph user on this standalone system, and
> I don’t think I’ll be able to control the user account of the monitoring
> hosts either (I think they might run under a user account dedicated to the
> monitoring) but I’m interested to what you think I should test here?
>
> I can definitely run the code as the root user, it can read my custom
> configuration and key files, when I specify those providing the admin user
> credentials, it works as expected, but when I specify the monitoring
> credentials it errors with that ceph_init message.
>
> My script can open and read the configuration and key files (I print those
> to screen), and I do attempt to pull back the environment before I execute
> the mount, and it does include my addition of the CEPH_ARGS. That said,
> those also work when those particular files are for the admin ceph account,
> and that can’t be picking up anything from the default locations as I’ve
> deliberately removed them from there.
>
> Is there any way to make the Python LibCephFS more verbose to better
> understand its error message?
>
> Adam.
>
> From: Bogdan Adrian Velica 
> Sent: 25 July 2022 14:50
> To: Adam Carrgilson (NBI) 
> Cc: ceph-users@ceph.io
> Subject: Re: [ceph-users] LibCephFS Python Mount Failure
>
> Hi Adam,
>
> I think this might be related to the user you are running the script as,
> try running the scrip as ceph user (or the user you are running your ceph
> with). Also make sure the variable os.environ.get is used (i might be
> mistaking here). do a print or something first to see the key is loaded.
> Just my 2 cents...
>
> Best of luck,
>
>  --
> Bogdan Velica
> Ceph Support Engineer
>
> croit GmbH, Freseniusstr. 31h, 81247 Munich
> 
> Com. register: Amtsgericht Munich HRB 231263
> Web: https://croit.io
>
> On Mon, Jul 25, 2022 at 4:06 PM Adam Carrgilson (NBI) <
> adam.carrgil...@nbi.ac.uk> wrote:
> Hi all,
>
> I'm trying to put together a script to gather CephFS quota utilisation.
> I'm using the CephFS Python library from here:
> https://docs.ceph.com/en/latest/cephfs/api/libcephfs-py/
> and I've followed the rather a good guide on how to use it here:
> https://jayjeetc.medium.com/up-and-running-with-libcephfs-7629455f0cdc#934a
>
> I have been able to get this working, however; I want this to be able to
> be portable to run it on our monitoring agents, and specifically, I want to
> be able to use a limited permission account, so read-only permissions and
> network limitations.
> I originally couldn't find a method to specify a custom keyfile to use
> through the library, but with some assistance, I've found that I can use
> the Python command: os.env

[ceph-users] Deletion of master branch July 28

[David Galloway]

Hi all,

I slowly worked my way through re-targeting any lingering ceph.git PRs 
(there were 300+ of them) from the master to main branch.  There were a 
few dozen repos I wanted to rename the master branch on and the tool I 
used did not automatically retarget existing PRs.


This means the time has come to delete the master branch from ceph.git. 
 Please update any automation you have that points to the master branch.


Thanks,
--
David Galloway
Principal Systems Administrator
Ceph Engineering

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: weird performance issue on ceph

[Hans van den Bogert]

Is rook/CSI still not using efficient rbd object maps ?

It could be that you issued a new benchmark while ceph was busy
(inefficiently) removing the  old rbd images. This is quite a stretch but
could be worth exploring.

On Mon, Jul 25, 2022, 21:42 Mark Nelson  wrote:

> I don't think so if this is just plain old RBD.  RBD  shouldn't require
> a bunch of RocksDB iterator seeks in the read/write hot path and writes
> should pretty quickly clear out tombstones as part of the memtable flush
> and compaction process even in the slow case.  Maybe in some kind of
> pathologically bad read-only corner case with no onode cache but it
> would be bad for more reasons than what's happening in that tracker
> ticket imho (even reading onodes from rocksdb block cache is
> significantly slower than BlueStore's onode cache).
>
> If RBD mirror (or snapshots) are involved that could be a different
> story though.  I believe to deal with deletes in that case we have to go
> through iteration/deletion loops that have same root issue as what's
> going on in the tracker ticket and it can end up impacting client IO.
> Gabi and Paul and testing/reworking how the snapmapper works and I've
> started a sort of a catch-all PR for improving our RocksDB tunings/glue
> here:
>
>
> https://github.com/ceph/ceph/pull/47221
>
>
> Mark
>
> On 7/25/22 12:48, Frank Schilder wrote:
> > Could it be related to this performance death trap:
> https://tracker.ceph.com/issues/55324 ?
> > =
> > Frank Schilder
> > AIT Risø Campus
> > Bygning 109, rum S14
> >
> > 
> > From: Mark Nelson 
> > Sent: 25 July 2022 18:50
> > To: ceph-users@ceph.io
> > Subject: [ceph-users] Re: weird performance issue on ceph
> >
> > Hi Zoltan,
> >
> >
> > We have a very similar setup with one of our upstream community
> > performance test clusters.  60 4TB PM983 drives spread across 10 nodes.
> > We get similar numbers to what you are initially seeing (scaled down to
> > 60 drives) though with somewhat lower random read IOPS (we tend to max
> > out at around 2M with 60 drives on this HW). I haven't seen any issues
> > with quincy like what you are describing, but on this cluster most of
> > the tests have been on bare metal.  One issue we have noticed with the
> > PM983 drives is that they may be more susceptible to non-optimal write
> > patterns causing slowdowns vs other NVMe drives in the lab.  We actually
> > had to issue a last minute PR for quincy to change the disk allocation
> > behavior to deal with it.  See:
> >
> >
> > https://github.com/ceph/ceph/pull/45771
> >
> > https://github.com/ceph/ceph/pull/45884
> >
> >
> > I don't *think* this is the issue you are hitting since the fix in
> > #45884 should have taken care of it, but it might be something to keep
> > in the back of your mind.  Otherwise, the fact that you are seeing such
> > a dramatic difference across both small and large read/write benchmarks
> > makes me think there is something else going on.  Is there any chance
> > that some other bottleneck is being imposed when the pods and volumes
> > are deleted and recreated? Might be worth looking at memory and CPU
> > usage of the OSDs in all of the cases and RocksDB flushing/compaction
> > stats from the OSD logs.  Also a quick check with collectl/iostat/sar
> > during the slow case to make sure none of the drives are showing latency
> > and built up IOs in the device queues.
> >
> > If you want to go deeper down the rabbit hole you can try running my
> > wallclock profiler against one of your OSDs in the fast/slow cases, but
> > you'll have to make sure it has access to debug symbols:
> >
> >
> > https://github.com/markhpc/uwpmp.git
> >
> >
> > run it like:
> >
> >
> > ./uwpmp -n 1 -p  -b libdw > output.txt
> >
> >
> > If the libdw backend is having problems you can use -b libdwarf instead,
> > but it's much slower and takes longer to collect as many samples (you
> > might want to do -n 1000 instead).
> >
> >
> > Mark
> >
> >
> > On 7/25/22 11:17, Zoltan Langi wrote:
> >> Hi people, we got an interesting issue here and I would like to ask if
> >> anyone seen anything like this before.
> >>
> >>
> >> First: our system:
> >>
> >> The ceph version is 17.2.1 but we also seen the same behaviour on
> 16.2.9.
> >>
> >> Our kernel version is 5.13.0-51 and our NVMe disks are Samsung PM983.
> >>
> >> In our deployment we got 12 nodes in total, 72 disks and 2 osd per
> >> disk makes 144 osd in total.
> >>
> >> The depoyment was done by ceph-rook with default values, 6 CPU cores
> >> allocated to the OSD each and 4GB of memory allocated to each OSD.
> >>
> >>
> >> The issue we are experiencing: We create for example 100 volumes via
> >> ceph-csi and attach it to kubernetes pods via rbd. We talk about 100
> >> volumes in total, 2GB each. We run fio performance tests (read, write,
> >> mixed) on them so the volumes are being used heavily. Ceph delivers
> >> good performance, no problems as all.
> >>
> >> Perfo

[ceph-users] Re: weird performance issue on ceph

[Marc]

Afaik is csi just some go code that maps an rbd image, it does as you would do 
it from the command line. Then again they really do not understand csi there, 
and are just developing a kubernetes 'driver'.

> 
> Is rook/CSI still not using efficient rbd object maps ?
> 
> It could be that you issued a new benchmark while ceph was busy
> (inefficiently) removing the  old rbd images. This is quite a stretch
> but
> could be worth exploring.
> 
> On Mon, Jul 25, 2022, 21:42 Mark Nelson  wrote:
> 
> > I don't think so if this is just plain old RBD.  RBD  shouldn't
> require
> > a bunch of RocksDB iterator seeks in the read/write hot path and
> writes
> > should pretty quickly clear out tombstones as part of the memtable
> flush
> > and compaction process even in the slow case.  Maybe in some kind of
> > pathologically bad read-only corner case with no onode cache but it
> > would be bad for more reasons than what's happening in that tracker
> > ticket imho (even reading onodes from rocksdb block cache is
> > significantly slower than BlueStore's onode cache).
> >
> > If RBD mirror (or snapshots) are involved that could be a different
> > story though.  I believe to deal with deletes in that case we have to
> go
> > through iteration/deletion loops that have same root issue as what's
> > going on in the tracker ticket and it can end up impacting client IO.
> > Gabi and Paul and testing/reworking how the snapmapper works and I've
> > started a sort of a catch-all PR for improving our RocksDB
> tunings/glue
> > here:
> >
> >
> > https://github.com/ceph/ceph/pull/47221
> >
> >
> > Mark
> >
> > On 7/25/22 12:48, Frank Schilder wrote:
> > > Could it be related to this performance death trap:
> > https://tracker.ceph.com/issues/55324 ?
> > > =
> > > Frank Schilder
> > > AIT Risø Campus
> > > Bygning 109, rum S14
> > >
> > > 
> > > From: Mark Nelson 
> > > Sent: 25 July 2022 18:50
> > > To: ceph-users@ceph.io
> > > Subject: [ceph-users] Re: weird performance issue on ceph
> > >
> > > Hi Zoltan,
> > >
> > >
> > > We have a very similar setup with one of our upstream community
> > > performance test clusters.  60 4TB PM983 drives spread across 10
> nodes.
> > > We get similar numbers to what you are initially seeing (scaled down
> to
> > > 60 drives) though with somewhat lower random read IOPS (we tend to
> max
> > > out at around 2M with 60 drives on this HW). I haven't seen any
> issues
> > > with quincy like what you are describing, but on this cluster most
> of
> > > the tests have been on bare metal.  One issue we have noticed with
> the
> > > PM983 drives is that they may be more susceptible to non-optimal
> write
> > > patterns causing slowdowns vs other NVMe drives in the lab.  We
> actually
> > > had to issue a last minute PR for quincy to change the disk
> allocation
> > > behavior to deal with it.  See:
> > >
> > >
> > > https://github.com/ceph/ceph/pull/45771
> > >
> > > https://github.com/ceph/ceph/pull/45884
> > >
> > >
> > > I don't *think* this is the issue you are hitting since the fix in
> > > #45884 should have taken care of it, but it might be something to
> keep
> > > in the back of your mind.  Otherwise, the fact that you are seeing
> such
> > > a dramatic difference across both small and large read/write
> benchmarks
> > > makes me think there is something else going on.  Is there any
> chance
> > > that some other bottleneck is being imposed when the pods and
> volumes
> > > are deleted and recreated? Might be worth looking at memory and CPU
> > > usage of the OSDs in all of the cases and RocksDB
> flushing/compaction
> > > stats from the OSD logs.  Also a quick check with
> collectl/iostat/sar
> > > during the slow case to make sure none of the drives are showing
> latency
> > > and built up IOs in the device queues.
> > >
> > > If you want to go deeper down the rabbit hole you can try running my
> > > wallclock profiler against one of your OSDs in the fast/slow cases,
> but
> > > you'll have to make sure it has access to debug symbols:
> > >
> > >
> > > https://github.com/markhpc/uwpmp.git
> > >
> > >
> > > run it like:
> > >
> > >
> > > ./uwpmp -n 1 -p  -b libdw > output.txt
> > >
> > >
> > > If the libdw backend is having problems you can use -b libdwarf
> instead,
> > > but it's much slower and takes longer to collect as many samples
> (you
> > > might want to do -n 1000 instead).
> > >
> > >
> > > Mark
> > >
> > >
> > > On 7/25/22 11:17, Zoltan Langi wrote:
> > >> Hi people, we got an interesting issue here and I would like to ask
> if
> > >> anyone seen anything like this before.
> > >>
> > >>
> > >> First: our system:
> > >>
> > >> The ceph version is 17.2.1 but we also seen the same behaviour on
> > 16.2.9.
> > >>
> > >> Our kernel version is 5.13.0-51 and our NVMe disks are Samsung
> PM983.
> > >>
> > >> In our deployment we got 12 nodes in total, 72 disks and 2 osd per
> > >> disk makes 144 osd in total.
> > >>
> > >> The depo

[ceph-users] Re: octopus v15.2.17 QE Validation status

[Ramana Krisna Venkatesh Raja]

On Thu, Jul 21, 2022 at 10:28 AM Yuri Weinstein  wrote:
>
> Details of this release are summarized here:
>
> https://tracker.ceph.com/issues/56484
> Release Notes - https://github.com/ceph/ceph/pull/47198
>
> Seeking approvals for:
>
> rados - Neha, Travis, Ernesto, Adam
> rgw - Casey
> fs, kcephfs, multimds - Venky, Patrick

fs, kcephfs, multimds approved.

Review of results,
https://tracker.ceph.com/projects/cephfs/wiki/Octopus#2022-Jul-26

Thanks,
Ramana

> rbd - Ilya, Deepika
> krbd  Ilya, Deepika
> ceph-ansible - Brad pls take a look
>
> Please reply to this email with approval and/or trackers of known issues/PRs 
> to address them.
>
> PS:  some tests are still running but I will be off-line for several hours 
> and wanted to start the review process.
>
> Thx
> YuriW
> ___
> Dev mailing list -- d...@ceph.io
> To unsubscribe send an email to dev-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: octopus v15.2.17 QE Validation status

[Gregory Farnum]

We can’t do the final release until the recent mgr/volumes security fixes
get merged in, though.
https://github.com/ceph/ceph/pull/47236

On Tue, Jul 26, 2022 at 3:12 PM Ramana Krisna Venkatesh Raja <
rr...@redhat.com> wrote:

> On Thu, Jul 21, 2022 at 10:28 AM Yuri Weinstein 
> wrote:
> >
> > Details of this release are summarized here:
> >
> > https://tracker.ceph.com/issues/56484
> > Release Notes - https://github.com/ceph/ceph/pull/47198
> >
> > Seeking approvals for:
> >
> > rados - Neha, Travis, Ernesto, Adam
> > rgw - Casey
> > fs, kcephfs, multimds - Venky, Patrick
>
> fs, kcephfs, multimds approved.
>
> Review of results,
> https://tracker.ceph.com/projects/cephfs/wiki/Octopus#2022-Jul-26
>
> Thanks,
> Ramana
>
> > rbd - Ilya, Deepika
> > krbd  Ilya, Deepika
> > ceph-ansible - Brad pls take a look
> >
> > Please reply to this email with approval and/or trackers of known
> issues/PRs to address them.
> >
> > PS:  some tests are still running but I will be off-line for several
> hours and wanted to start the review process.
> >
> > Thx
> > YuriW
> > ___
> > Dev mailing list -- d...@ceph.io
> > To unsubscribe send an email to dev-le...@ceph.io
>
> ___
> Dev mailing list -- d...@ceph.io
> To unsubscribe send an email to dev-le...@ceph.io
>
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: octopus v15.2.17 QE Validation status

[Gregory Farnum]

On Tue, Jul 26, 2022 at 3:41 PM Yuri Weinstein  wrote:

> Greg, I started testing this PR.
> What do you want to rerun for it?  Are fs, kcephfs, multimds suites
> sufficient?


We just need to run the mgr/volumes tests — I think those are all in the fs
suite but Kotresh or Ramana can let us know.
-Greg


>
> On Tue, Jul 26, 2022 at 3:16 PM Gregory Farnum  wrote:
> >
> > We can’t do the final release until the recent mgr/volumes security
> fixes get merged in, though.
> > https://github.com/ceph/ceph/pull/47236
> >
> > On Tue, Jul 26, 2022 at 3:12 PM Ramana Krisna Venkatesh Raja <
> rr...@redhat.com> wrote:
> >>
> >> On Thu, Jul 21, 2022 at 10:28 AM Yuri Weinstein 
> wrote:
> >> >
> >> > Details of this release are summarized here:
> >> >
> >> > https://tracker.ceph.com/issues/56484
> >> > Release Notes - https://github.com/ceph/ceph/pull/47198
> >> >
> >> > Seeking approvals for:
> >> >
> >> > rados - Neha, Travis, Ernesto, Adam
> >> > rgw - Casey
> >> > fs, kcephfs, multimds - Venky, Patrick
> >>
> >> fs, kcephfs, multimds approved.
> >>
> >> Review of results,
> >> https://tracker.ceph.com/projects/cephfs/wiki/Octopus#2022-Jul-26
> >>
> >> Thanks,
> >> Ramana
> >>
> >> > rbd - Ilya, Deepika
> >> > krbd  Ilya, Deepika
> >> > ceph-ansible - Brad pls take a look
> >> >
> >> > Please reply to this email with approval and/or trackers of known
> issues/PRs to address them.
> >> >
> >> > PS:  some tests are still running but I will be off-line for several
> hours and wanted to start the review process.
> >> >
> >> > Thx
> >> > YuriW
> >> > ___
> >> > Dev mailing list -- d...@ceph.io
> >> > To unsubscribe send an email to dev-le...@ceph.io
> >>
> >> ___
> >> Dev mailing list -- d...@ceph.io
> >> To unsubscribe send an email to dev-le...@ceph.io
> >>
>
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: octopus v15.2.17 QE Validation status

[Kotresh Hiremath Ravishankar]

On Wed, Jul 27, 2022 at 5:02 AM Gregory Farnum  wrote:

> On Tue, Jul 26, 2022 at 3:41 PM Yuri Weinstein 
> wrote:
>
>> Greg, I started testing this PR.
>> What do you want to rerun for it?  Are fs, kcephfs, multimds suites
>> sufficient?
>
>
> We just need to run the mgr/volumes tests — I think those are all in the
> fs suite but Kotresh or Ramana can let us know.
> -Greg
>

Right. Running fs:volumes suite would be sufficient.

>
>
>>
>> On Tue, Jul 26, 2022 at 3:16 PM Gregory Farnum 
>> wrote:
>> >
>> > We can’t do the final release until the recent mgr/volumes security
>> fixes get merged in, though.
>> > https://github.com/ceph/ceph/pull/47236
>> >
>> > On Tue, Jul 26, 2022 at 3:12 PM Ramana Krisna Venkatesh Raja <
>> rr...@redhat.com> wrote:
>> >>
>> >> On Thu, Jul 21, 2022 at 10:28 AM Yuri Weinstein 
>> wrote:
>> >> >
>> >> > Details of this release are summarized here:
>> >> >
>> >> > https://tracker.ceph.com/issues/56484
>> >> > Release Notes - https://github.com/ceph/ceph/pull/47198
>> >> >
>> >> > Seeking approvals for:
>> >> >
>> >> > rados - Neha, Travis, Ernesto, Adam
>> >> > rgw - Casey
>> >> > fs, kcephfs, multimds - Venky, Patrick
>> >>
>> >> fs, kcephfs, multimds approved.
>> >>
>> >> Review of results,
>> >> https://tracker.ceph.com/projects/cephfs/wiki/Octopus#2022-Jul-26
>> >>
>> >> Thanks,
>> >> Ramana
>> >>
>> >> > rbd - Ilya, Deepika
>> >> > krbd  Ilya, Deepika
>> >> > ceph-ansible - Brad pls take a look
>> >> >
>> >> > Please reply to this email with approval and/or trackers of known
>> issues/PRs to address them.
>> >> >
>> >> > PS:  some tests are still running but I will be off-line for several
>> hours and wanted to start the review process.
>> >> >
>> >> > Thx
>> >> > YuriW
>> >> > ___
>> >> > Dev mailing list -- d...@ceph.io
>> >> > To unsubscribe send an email to dev-le...@ceph.io
>> >>
>> >> ___
>> >> Dev mailing list -- d...@ceph.io
>> >> To unsubscribe send an email to dev-le...@ceph.io
>> >>
>>
>>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Ceph on RHEL 9

[Massimo Sgaravatto]

Dear all

Are there any updates on this question ?

In particular, since I am planning how to update my infrastructure, I would
be interested to know if there are plans to provide packages for centos9
stream for Pacific and/or Quincy

Thanks, Massimo


On Fri, Jun 10, 2022 at 6:46 PM Gregory Farnum  wrote:

> We aren't building for Centos 9 yet, so I guess the python dependency
> declarations don't work with the versions in that release.
> I've put updating to 9 on the agenda for the next CLT.
>
> (Do note that we don't test upstream packages against RHEL, so if
> Centos Stream does something which doesn't match the RHEL release it
> still might get busted.)
> -Greg
>
> On Thu, Jun 9, 2022 at 6:57 PM Robert W. Eckert 
> wrote:
> >
> > Does anyone have any pointers to install CEPH on Rhel 9?
> >
> > -Original Message-
> > From: Robert W. Eckert 
> > Sent: Saturday, May 28, 2022 8:28 PM
> > To: ceph-users@ceph.io
> > Subject: [ceph-users] Ceph on RHEL 9
> >
> > Hi- I started to update my 3 host cluster to RHEL 9, but came across a
> bit of a stumbling block.
> >
> > The upgrade process uses the RHEL leapp process, which ran through a few
> simple things to clean up, and told me everything was hunky dory, but when
> I kicked off the first server, the server wouldn't boot because I had a
> ceph filesystem mounted in /etc/fstab, commenting it out, let the upgrade
> happen.
> >
> > Then I went to check on the ceph client which appears to be uninstalled.
> >
> > When I tried to install ceph,  I got:
> >
> > [root@story ~]# dnf install ceph
> > Updating Subscription Management repositories.
> > Last metadata expiration check: 0:07:58 ago on Sat 28 May 2022 08:06:52
> PM EDT.
> > Error:
> > Problem: package ceph-2:17.2.0-0.el8.x86_64 requires ceph-mgr =
> 2:17.2.0-0.el8, but none of the providers can be installed
> >   - conflicting requests
> >   - nothing provides libpython3.6m.so.1.0()(64bit) needed by
> ceph-mgr-2:17.2.0-0.el8.x86_64 (try to add '--skip-broken' to skip
> uninstallable packages or '--nobest' to use not only best candidate
> packages)
> >
> > This is the content of my /etc/yum.repos.d/ceph.conf
> >
> > [ceph]
> > name=Ceph packages for $basearch
> > baseurl=https://download.ceph.com/rpm-quincy/el8/$basearch
> > enabled=1
> > priority=2
> > gpgcheck=1
> > gpgkey=https://download.ceph.com/keys/release.asc
> >
> > [ceph-noarch]
> > name=Ceph noarch packages
> > baseurl=https://download.ceph.com/rpm-quincy/el8/noarch
> > enabled=1
> > priority=2
> > gpgcheck=1
> > gpgkey=https://download.ceph.com/keys/release.asc
> >
> > [ceph-source]
> > name=Ceph source packages
> > baseurl=https://download.ceph.com/rpm-quincy/el8/SRPMS
> > enabled=0
> > priority=2
> > gpgcheck=1
> > gpgkey=https://download.ceph.com/keys/release.asc
> > Is there anything I should change for el9 (I don't see el9 rpms out yet).
> >
> > Or should I  wait before updating the other two servers?
> >
> > Thanks,
> > Rob
> >
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an
> email to ceph-users-le...@ceph.io
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
> >
>
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io