[ceph-users] Re: squid 19.2.1 RC QE validation status

[Christian Rohmann]

Hey Adam,

On 11.01.25 12:52 AM, Adam Emerson wrote:

On 10/01/2025, Yuri Weinstein wrote:

This PR https://github.com/ceph/ceph/pull/61306 was cherry-picked
Adam, pls see the run for the Build 4

Laura, Adam approves rgw, we are ready for gibba and LRC/sepia upgrades.


I hereby approve the RGW run. Thanks and sorry for the last minute fix.



Is the  broken rgw_s3_auth_order https://tracker.ceph.com/issues/68393 
not relevant enough for the release then?

There is a PR open https://github.com/ceph/ceph/pull/61162

Also there are some desperate comments about this breaking / hindering 
multi-site sync (https://tracker.ceph.com/issues/69183#note-4)

which I totally agree with.



Regards


Christian



___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Many misplaced PG's, full OSD's and a good amount of manual intervention to keep my Ceph cluster alive.

[Anthony D'Atri]

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Many misplaced PG's, full OSD's and a good amount of manual intervention to keep my Ceph cluster alive.

[Bruno Gomes Pessanha]

Hi everyone. Yes. All the tips definitely helped! Now I have more free
space in the pools, the number of misplaced PG's decreased a lot and lower
std deviation of the usage of OSD's. The storage looks way healthier now.
Thanks a bunch!

I'm only confused by the number of misplaced PG's which never goes
below 5%. Every time it hits 5% it goes up and down like shown in this
quite interesting graph:
[image: image.png]

Any idea why that might be?

I had the impression that it might be related to the autobalancer that
kicks in and pg's are misplaced again. Or am I missing something?

Bruno

On Mon, 6 Jan 2025 at 16:00, Bruno Gomes Pessanha 
wrote:

> So you might set the full ratio to .98, backfillfull to .96.  Nearfull is
>> only cosmetic.
>
> Thanks for the advice. It seems to be working with 0.92 for now. If it
> gets stuck I'll increase it.
>
> On Mon, 6 Jan 2025 at 00:24, Anthony D'Atri 
> wrote:
>
>>
>>
>> Very solid advice here - that’s the beauty of Ceph community.
>>
>> Just adding to what Anthony mentioned: a reweight from 1 to 0.2 (and
>> back) is quite extreme and the cluster won’t like it.
>>
>>
>> And these days with the balancer, pg-upmap entries to the same effect are
>> a better idea.
>>
>> From the clients perspective Your main concern now is to keep the pools
>> “alive” with enough space while the backfilling takes place.
>>
>>
>> To that end, you can *temporarily* give yourself a bit more margin:
>>
>> ceph osd set-nearfull-ratio .85
>> ceph osd set-backfillfull-ratio .90
>> ceph osd set-full-ratio .95
>>
>> Those are the default values, and Ceph (now) enforces that the values are
>> >= (or maybe >) in that order.
>>
>> So you might set the full ratio to .98, backfillfull to .96.  Nearfull is
>> only cosmetic.
>>
>> But absolutely do not forget to revert to default values once the cluster
>> is balanced, or to other values that you make an educated decision to
>> choose.
>>
>> Even with plenty of OSDs that are not filled you might hit a single
>> overfilled OSD and the whole pool will stop accepting new data.
>>
>>
>> Yep, see above.  Not immediately clear to me why that data pool is so
>> full unless the CRUSH rule / device classes are wonky.
>>
>> Clients will start getting “No more space available” errors. That
>> happened to us with CephFS recently with a very similar scenario where the
>> cluster got much more data than expected in a short amount of time, not
>> fun.
>> With the balancer not working due to too many misplaced objects that’s an
>> increased risk so just heads up and keep that in mind. To get things
>> working we simply balanced manually the OSDs with upmaps moving data from
>> the most full ones to the least full ones (our builtin balancer sadly
>> does not work).
>>
>>
>> One small observation:
>> I’ve noticed that 'ceph osd pool ls detail |grep cephfs.cephfs01.data’
>> has pg_num increased but the pgp_num is still the same.
>> You will need to set it as well for data migration to new pgs to happen:
>> https://docs.ceph.com/en/mimic/rados/operations/placement-groups/#set-the-number-of-placement-groups
>>
>>
>> The mgr usually does that for recent Ceph releases.  With older releases
>> we had to incremental pg_num and pgp_num in lockstep, which was kind of a
>> pain.
>>
>>
>>
>> Best,
>>
>> *Laimis J.*
>>
>> On 5 Jan 2025, at 16:11, Anthony D'Atri  wrote:
>>
>>
>> What reweighs have been set for the top OSDs (ceph osd df tree)?
>>
>> Right now they are all at 1.0. I had to lower them to something close to
>> 0.2 in order to free up space but I changed them back to 1.0. Should I
>> lower them while the backfill is happening?
>>
>>
>> Old-style legacy override reweights don’t mesh well with the balancer.
>>   Best to leave them at 1.00.
>>
>> 0.2 is pretty extreme, back in the day I rarely went below 0.8.
>>
>> ```
>> "optimize_result": "Too many objects (0.355160 > 0.05) are misplaced;
>> try again late
>> ```
>>
>>
>> That should clear.  The balancer doesn’t want to stir up trouble if the
>> cluster already has a bunch of backfill / recovery going on.  Patience!
>>
>> default.rgw.buckets.data10  1024  197 TiB  133.75M  592 TiB  93.69
>>   13 TiB
>> default.rgw.buckets.non-ec  1132   78 MiB1.43M   17 GiB
>>
>>
>> That’s odd that the data pool is that full but the others aren’t.
>>
>> Please send `ceph osd crush rule dump `.  And `ceph osd dump | grep pool`
>>
>>
>>
>> I also tried changing the following but it does not seem to persist:
>>
>>
>> Could be an mclock thing.
>>
>> 1. Why I ended up with so many misplaced PG's since there were no changes
>> on the cluster: number of osd's, hosts, etc.
>>
>>
>> Probably a result of the autoscaler splitting PGs or of some change to
>> CRUSH rules such that some data can’t be placed.
>>
>> 2. Is it ok to change the target_max_misplaced_ratio to something higher
>> than .05 so the autobalancer would work and I wouldn't have to constantly
>> rebalance the osd's manually?
>>
>>
>> I wouldn’t, that’s a symptom not the 

[ceph-users] issue with new AWS cli when upload: MissingContentLength

[Szabo, Istvan (Agoda)]

Hi,

Amazon released a new version of their cli today 
https://github.com/aws/aws-cli/tags and seems to break our stuffs with the 
following error when PUT object happens:

bash-4.2# /usr/local/bin/aws --endpoint=https://endpoint --no-verify-ssl s3 cp 
online.txt s3://bucket/
upload failed: ./online.txt to s3://bucket/online.txt An error occurred 
(MissingContentLength) when calling the PutObject operation: Unknown

Wonder is there anything on ceph side that I can do to eliminate this?
The amazon workaround response_checksum_validation = when_required did not work.
https://github.com/aws/aws-cli/issues/9214

Thank you
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: squid 19.2.1 RC QE validation status

[Travis Nielsen]

When running the Rook CI against the latest squid devel image, we are
seeing issues creating OSDs, investigating with Guillaume...
https://github.com/rook/rook/issues/15282

Travis

On Wed, Jan 15, 2025 at 7:57 AM Laura Flores  wrote:

> The Gibba cluster has been upgraded.
>
> On Wed, Jan 15, 2025 at 7:27 AM Christian Rohmann <
> christian.rohm...@inovex.de> wrote:
>
> > Hey Adam,
> >
> > On 11.01.25 12:52 AM, Adam Emerson wrote:
> > > On 10/01/2025, Yuri Weinstein wrote:
> > >> This PR https://github.com/ceph/ceph/pull/61306 was cherry-picked
> > >> Adam, pls see the run for the Build 4
> > >>
> > >> Laura, Adam approves rgw, we are ready for gibba and LRC/sepia
> upgrades.
> > >
> > > I hereby approve the RGW run. Thanks and sorry for the last minute fix.
> >
> >
> > Is the  broken rgw_s3_auth_order https://tracker.ceph.com/issues/68393
> > not relevant enough for the release then?
> > There is a PR open https://github.com/ceph/ceph/pull/61162
> >
> > Also there are some desperate comments about this breaking / hindering
> > multi-site sync (https://tracker.ceph.com/issues/69183#note-4)
> > which I totally agree with.
> >
> >
> >
> > Regards
> >
> >
> > Christian
> >
> >
> >
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
> >
>
>
> --
>
> Laura Flores
>
> She/Her/Hers
>
> Software Engineer, Ceph Storage 
>
> Chicago, IL
>
> lflo...@ibm.com | lflo...@redhat.com 
> M: +17087388804
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: squid 19.2.1 RC QE validation status

[Laura Flores]

The Gibba cluster has been upgraded.

On Wed, Jan 15, 2025 at 7:27 AM Christian Rohmann <
christian.rohm...@inovex.de> wrote:

> Hey Adam,
>
> On 11.01.25 12:52 AM, Adam Emerson wrote:
> > On 10/01/2025, Yuri Weinstein wrote:
> >> This PR https://github.com/ceph/ceph/pull/61306 was cherry-picked
> >> Adam, pls see the run for the Build 4
> >>
> >> Laura, Adam approves rgw, we are ready for gibba and LRC/sepia upgrades.
> >
> > I hereby approve the RGW run. Thanks and sorry for the last minute fix.
>
>
> Is the  broken rgw_s3_auth_order https://tracker.ceph.com/issues/68393
> not relevant enough for the release then?
> There is a PR open https://github.com/ceph/ceph/pull/61162
>
> Also there are some desperate comments about this breaking / hindering
> multi-site sync (https://tracker.ceph.com/issues/69183#note-4)
> which I totally agree with.
>
>
>
> Regards
>
>
> Christian
>
>
>
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>


-- 

Laura Flores

She/Her/Hers

Software Engineer, Ceph Storage 

Chicago, IL

lflo...@ibm.com | lflo...@redhat.com 
M: +17087388804
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Help needed, ceph fs down due to large stray dir

[Frank Schilder]

Dear all,

we finally managed to collect perf data and it seems to show a smoking gun. 
Since this thread is already heavily cluttered on lists.ceph.io I started a new 
one: "MDS hung in purge_stale_snap_data after populating cache" 
(https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/XLYHTZBD4AXNPFZOLG7NG24Z4DWXIARG/).

Please take a look at this post.

Thanks and best regards!
=
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14


From: Frédéric Nass 
Sent: Tuesday, January 14, 2025 8:25 AM
To: Frank Schilder; ceph-users@ceph.io
Cc: Dan van der Ster; Patrick Donnelly; Bailey Allison; Spencer Macphee
Subject: Re: [ceph-users] Re: Help needed, ceph fs down due to large stray dir

Hi Frank,

More than ever. You should open a tracker and post debug logs there so anyone 
can have a look.

Regards,
Frédéric.

De : Frank Schilder 
Envoyé : lundi 13 janvier 2025 17:39
À : ceph-users@ceph.io
Cc: Dan van der Ster; Patrick Donnelly; Bailey Allison; Spencer Macphee
Objet : [ceph-users] Re: Help needed, ceph fs down due to large stray dir


Dear all,

a quick update and some answers. We set up a dedicated host for running an MDS 
and debugging the problem. On this host we have 750G RAM, 4T swap and 4T log, 
both on fast SSDs. Plan is to monitor with "perf top" the MDS becoming the 
designated MDS for the problematic rank and also pull out a detailed log about 
the startup until the MDS hangs. I have some questions about that, a new 
observation that might be relevant and some answers to some suggestions, in 
that order.

I need to install debug info for perf to give useful output. I can't find a 
meta package that has all ceph-debuginfo rpms as a dependency. Which ones 
should I install? Also, should I install some kernel debug info packages? 
Please note that every restart takes about 1h to hit the issue. I would like to 
have as much as possible installed the first time.

A new observation: After every restart cycle the rank loads a little bit less 
into cache. However, the num_stray count does not decrease. Could that mean the 
problem is not the high num_stray count but something else?

Answer to a suggestion: It is not possible to access anything on the MDS or the 
entire file system. Hence, trying to stat some files/dirs is not possible. 
Furthermore, not all stray items can be reintegrated with this method and I'm 
afraid our stray items are mostly of this nature. This means that in octopus 
the only way to trim (evaluate) stray items was an MDS restart. For details, 
the relevant discussions are 
https://www.spinics.net/lists/ceph-users/msg70459.html and 
https://www.spinics.net/lists/ceph-users/msg73150.html with the most important 
info in this message: https://www.spinics.net/lists/ceph-users/msg70849.html .

Summary: As part of the debugging back then I executed a recursive stat-ing of 
files and directories on the *entire* file system only to observe that the 
stray count didn't change. This was when Gregory finally explained that hard 
links can block stray removal on snaptrim also for paths that are no longer 
accessible through the file system or any snapshots, that is, the usual stray 
evaluation doesn't have any effect. That's the situation we are in, we need the 
MDS do it itself.

A correction: It was actually Venky Shankar participating in this communication 
and not Neha. Is Venkhy still working on the FS?

Thanks for package hints and best regards,
=
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] MDS hung in purge_stale_snap_data after populating cache

[Frank Schilder]

Hi all,

this post is related to "Help needed, ceph fs down due to large stray dir" 
(https://www.spinics.net/lists/ceph-users/msg85394.html).

We finally got the MDS up and could record what it is doing before and after 
getting hung. We opened tracker https://tracker.ceph.com/issues/69547 . The 
uploads were too large to attach. For a quick overview of the key findings see 
the 3 screen shots posted here: https://imgur.com/a/RF7ExSP .

Apparently, the MDS gets hung in CInode::purge_stale_snap_data after populating 
the cache. I hope that the data uploaded will give someone a clue what is 
happening and how we can get out of it.

Putting my hopes on tomorrow after sleeping a bit (well, at least trying to).

Thanks for all your help with this critical situation.

Best regards,
=
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: MDS crashing on startup

[Frank Schilder]

Hi Dan,

we finally managed to get everything up and collect debug info. Its ceph-posted 
since all files exceeded the limit for attachments. A quick overview of the 
most important findings is here: https://imgur.com/a/RF7ExSP. Please note that 
I started a new thread to reduce clutter: "MDS hung in purge_stale_snap_data 
after populating cache" 
(https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/XLYHTZBD4AXNPFZOLG7NG24Z4DWXIARG/).

A tracker is open at https://tracker.ceph.com/issues/69547

I hope there is a way out of this.

Thanks for helping and best regards.
=
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Ceph symbols for v15_2_0 in pacific libceph-common

[Frank Schilder]

Hi all,

during debugging of an MDS problem we observed something that looks odd. The 
output of perf top seems to show symbols from v15 (octopus) on a pacific (v16) 
installation:

  23.56%  ceph-mds  [.] std::_Rb_tree
   7.02%  libceph-common.so.2   [.] ceph::buffer::v15_2_0::ptr::copy_out
   4.99%  ceph-mds  [.] std::_Hashtable::copy
   2.53%  ceph-mds  [.] std::_Rb_tree::operator+=
   1.68%  ceph-mds  [.] MDCache::populate_mydir
   1.56%  ceph-mds  [.] std::_Rb_tree, std::_S
   1.38%  [kernel]  [k] clear_page_erms
   1.06%  [kernel]  [k] native_irq_return_iret

The pacific packages are installed from download.ceph.com and all package 
candidates claim to be v16:

# yum provides "/*/libceph-common.so.2"
Last metadata expiration check: 0:45:57 ago on Wed 15 Jan 2025 12:14:09 PM CET.
librados2-2:16.2.15-0.el8.x86_64 : RADOS distributed object store client library
Repo: @System
Matched from:
Filename: /usr/lib64/ceph/libceph-common.so.2

librados2-2:16.2.15-0.el8.x86_64 : RADOS distributed object store client library
Repo: ceph
Matched from:
Filename: /usr/lib64/ceph/libceph-common.so.2

Why do we see v15 symbols here or am I interpreting the symbol name 
ceph::buffer::v15_2_0::list::iterator_impl::copy incorrectly?

Thanks and best regards,
=
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Ceph User + Dev Meeting Information

[Laura Flores]

Hi all,

It was brought to my attention that some users were unaware that the old
User + Dev meetup day changed from Thursdays to Wednesdays.

To stay informed about future User + Dev meetings, you can:

1. Join the Meetup group here: https://www.meetup.com/ceph-user-group/,
which will allow you to RSVP to upcoming events.

2. Download the Ceph Community Calendar, which is accessible here:
https://ceph.io/en/community/meetups/

3. Follow updates about upcoming topics, etc. for these meetings in
the #ceph-user-dev-meetup Slack channel at this link:
https://ceph-storage.slack.com/archives/C072CRU63H9

Thanks,
Laura Flores

-- 

Laura Flores

She/Her/Hers

Software Engineer, Ceph Storage 

Chicago, IL

lflo...@ibm.com | lflo...@redhat.com 
M: +17087388804
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: squid 19.2.1 RC QE validation status

[Adam Emerson]

On 15/01/2025, Christian Rohmann wrote:
> Is the  broken rgw_s3_auth_order https://tracker.ceph.com/issues/68393 not
> relevant enough for the release then?
> There is a PR open https://github.com/ceph/ceph/pull/61162
> 
> Also there are some desperate comments about this breaking / hindering
> multi-site sync (https://tracker.ceph.com/issues/69183#note-4)
> which I totally agree with.

I apologize, but at this point, after talking to the team, blocking
the release would derail enough things for enough people I can't
really justify it.

I've added the fix to v18.2.5, the upcoming Reef, and 19.2.2, the
next squid, so it should be included in those.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io