[ceph-users] Re: reef 18.2.5 QE validation status

[Venky Shankar]

Hi Yuri,

On Tue, Apr 1, 2025 at 2:36 AM Yuri Weinstein  wrote:
>
> Release Notes -
>
> https://github.com/ceph/ceph/pull/62589
> https://github.com/ceph/ceph.io/pull/856
>
> Still seeking approvals/reviews for:
>
> rados - Travis?  Nizamudeen?
>
> rgw - Adam E approved?
>
> fs - Venky approved?

fs approved. Failures are

https://tracker.ceph.com/projects/cephfs/wiki/Reef#v1825

>
> upgrade-clients:client-upgrade-octopus-reef-reef - Ilya please take a look.
>
> upgrade/pacific-x (reef) - can this be deprecated?  Josh?  Neha?
>
> ceph-volume - Guillaume approved?
>
> On Thu, Mar 27, 2025 at 12:14 PM Laura Flores  wrote:
>
> > quincy-x approved:
> > https://tracker.ceph.com/projects/rados/wiki/REEF#v1825-httpstrackercephcomissues70563note-1-upgradequincy-x
> >
> > Asking Radek and Neha about pacific-x.
> >
> > On Thu, Mar 27, 2025 at 9:54 AM Yuri Weinstein 
> > wrote:
> >
> >> Venky, Guillaume pls review and approve fs and orch/cepadm
> >>
> >> Still awaiting arrivals:
> >>
> >> rados - Travis?  Nizamudeen? Adam King approved?
> >>
> >> rgw - Adam E approved?
> >>
> >> fs - Venky approved?
> >>
> >> upgrade-clients:client-upgrade-octopus-reef-reef - Ilya please take a
> >> look.  There are multiple runs
> >>
> >> upgrade/pacific-x (reef) - can this be deprecated?  Josh?  Neha?
> >> upgrade/quincy-x (reef) - Laura, Prashant please take a look.
> >>
> >> ceph-volume - Guillaume approved?
> >>
> >> On Wed, Mar 26, 2025 at 2:26 PM Yuri Weinstein 
> >> wrote:
> >> >
> >> > Ack, Travis
> >> > I was about to reply the same.
> >> >
> >> > Venky, Guillaume the PRs below were cherry-picked
> >> > I will rerun the fs and ceph-volume tests when the build is done
> >> >
> >> > https://github.com/ceph/ceph/pull/62492/commits
> >> > https://github.com/ceph/ceph/pull/62178/commits
> >> >
> >> > On Wed, Mar 26, 2025 at 2:20 PM Travis Nielsen 
> >> wrote:
> >> > >
> >> > > Oh sorry, forget my last email, thanks Laura for pointing out the
> >> obvious that this is for reef, not squid!
> >> > >
> >> > > On Wed, Mar 26, 2025 at 2:46 PM Travis Nielsen 
> >> wrote:
> >> > >>
> >> > >> Yuri, as of when did 18.2.5 include the latest squid branch? If [1]
> >> is included in 18.2.5, then we really need [2] merged before release, as it
> >> would be blocking Rook.
> >> > >>
> >> > >> [1] https://github.com/ceph/ceph/pull/62095 (merged to squid on
> >> March 19)
> >> > >> [2] https://tracker.ceph.com/issues/70667
> >> > >>
> >> > >> Thanks!
> >> > >> Travis
> >> > >>
> >> > >> On Wed, Mar 26, 2025 at 2:09 PM Ilya Dryomov 
> >> wrote:
> >> > >>>
> >> > >>> On Mon, Mar 24, 2025 at 10:40 PM Yuri Weinstein <
> >> ywein...@redhat.com> wrote:
> >> > >>> >
> >> > >>> > Details of this release are summarized here:
> >> > >>> >
> >> > >>> > https://tracker.ceph.com/issues/70563#note-1
> >> > >>> > Release Notes - TBD
> >> > >>> > LRC upgrade - TBD
> >> > >>> >
> >> > >>> > Seeking approvals/reviews for:
> >> > >>> >
> >> > >>> > smoke - Laura approved?
> >> > >>> >
> >> > >>> > rados - Radek, Laura approved? Travis?  Nizamudeen? Adam King
> >> approved?
> >> > >>> >
> >> > >>> > rgw - Adam E approved?
> >> > >>> >
> >> > >>> > fs - Venky is fixing QA suite, will need to be added and rerun
> >> > >>> >
> >> > >>> > orch - Adam King approved?
> >> > >>> >
> >> > >>> > rbd - Ilya approved?
> >> > >>> > krbd - Ilya approved?
> >> > >>>
> >> > >>> Hi Yuri,
> >> > >>>
> >> > >>> rbd and krbd approved.
> >> > >>>
> >> > >>> > upgrade-clients:client-upgrade-octopus-reef-reef - Ilya please
> >> take a look.
> >> > >>>
> >> > >>> I don't recall seeing this before -- try rerunning it a couple of
> >> times?
> >> > >>>
> >> > >>> Thanks,
> >> > >>>
> >> > >>> Ilya
> >> > >>> ___
> >> > >>> ceph-users mailing list -- ceph-users@ceph.io
> >> > >>> To unsubscribe send an email to ceph-users-le...@ceph.io
> >> ___
> >> Dev mailing list -- d...@ceph.io
> >> To unsubscribe send an email to dev-le...@ceph.io
> >>
> >
> >
> > --
> >
> > Laura Flores
> >
> > She/Her/Hers
> >
> > Software Engineer, Ceph Storage 
> >
> > Chicago, IL
> >
> > lflo...@ibm.com | lflo...@redhat.com 
> > M: +17087388804
> >
> >
> >
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io



-- 
Cheers,
Venky
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Prometheus anomaly in Reef

[Tim Holloway]

Hi Eugen,

I never used a spec file before now. It was all done directly originally.

One thing that came up, however, is that my 14 year old motherboards 
seem to have been rejecting the extra disks I've been trying to add. 
That includes a spinning disk, a SATA SSD and even an M.2 PCI adapter. 
Which, unfortunately, my mobos only support PCIe 2 anyway.


So I'm going to be doing some offline hardware testing on a spare box 
and worry about the spec file some other day.


   Thanks for all the help!

   Tim

On 4/1/25 02:43, Eugen Block wrote:

Hi Tim,

I'm glad you sorted it out. But I'm wondering, did the prometheus spec 
file ever work? I had just assumed that it had since you wrote you had 
prometheus up and running before, so I didn't even question the 
"networks" parameter in there. Now that you say you only used the 
"--placement" parameter, I started to wonder. Since it's the only 
"extra" paramter in your spec file, I must assume that it's the root 
cause of the failures. That would probably easy to test... if you 
still have the patience. ;-)


Zitat von Tim Holloway :


Let me close this out by describing where I went afterwards.

I did a ceph orch apply with explicit --placement of 2 servers, one 
being the existing ceph08 and the other being the dell02 that was 
getting the "unknown service" errors.


This caused the dell02 machine to get a working prometheus and shut 
up about the error. The ceph08 machine reported an error. It claimed 
that apparently the prometheus port was already in use. As it was, 
since prometheus was already running. The deployer should properly 
have understood that and acted in a way that allowed the deployment 
to process without complaint.


To clear the port in use issue, I restarted prometheus on ceph08, 
then when it persisted did a "ceph mgr fail". That cleared all of the 
prometheus-related complaints and gave me a "HEALTH OK" status.


I wasn't brave enough to attempt the YAML version again, considering 
that's where the problem started. I also didn't attempt to try an 
"orch apply" that omitted any running service host, for fear it 
wouldn't remove the omitted host.


So the problem is fixed, but it took a lot of banging and hammering 
to make it work.


 Tim


On 3/28/25 19:12, Tim Holloway wrote:
Almost forgot to say. I switched out disks and got rid of the OSD 
errors. I actually found a third independent location, so it should 
be a lot more failure resistant now.


So now it's only the prometheus stuff that's still complaining. 
Everything else is happy.

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io



___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: reef 18.2.5 QE validation status

[Nizamudeen A]

dashboard approved on behalf of @Afreen Misbah  (since
she is on vacation)

Regards,
Nizam

On Tue, Apr 1, 2025 at 2:36 AM Yuri Weinstein  wrote:

> Release Notes -
>
> https://github.com/ceph/ceph/pull/62589
> https://github.com/ceph/ceph.io/pull/856
>
> Still seeking approvals/reviews for:
>
> rados - Travis?  Nizamudeen?
>
> rgw - Adam E approved?
>
> fs - Venky approved?
>
> upgrade-clients:client-upgrade-octopus-reef-reef - Ilya please take a look.
>
> upgrade/pacific-x (reef) - can this be deprecated?  Josh?  Neha?
>
> ceph-volume - Guillaume approved?
>
> On Thu, Mar 27, 2025 at 12:14 PM Laura Flores  wrote:
>
> > quincy-x approved:
> >
> https://tracker.ceph.com/projects/rados/wiki/REEF#v1825-httpstrackercephcomissues70563note-1-upgradequincy-x
> >
> > Asking Radek and Neha about pacific-x.
> >
> > On Thu, Mar 27, 2025 at 9:54 AM Yuri Weinstein 
> > wrote:
> >
> >> Venky, Guillaume pls review and approve fs and orch/cepadm
> >>
> >> Still awaiting arrivals:
> >>
> >> rados - Travis?  Nizamudeen? Adam King approved?
> >>
> >> rgw - Adam E approved?
> >>
> >> fs - Venky approved?
> >>
> >> upgrade-clients:client-upgrade-octopus-reef-reef - Ilya please take a
> >> look.  There are multiple runs
> >>
> >> upgrade/pacific-x (reef) - can this be deprecated?  Josh?  Neha?
> >> upgrade/quincy-x (reef) - Laura, Prashant please take a look.
> >>
> >> ceph-volume - Guillaume approved?
> >>
> >> On Wed, Mar 26, 2025 at 2:26 PM Yuri Weinstein 
> >> wrote:
> >> >
> >> > Ack, Travis
> >> > I was about to reply the same.
> >> >
> >> > Venky, Guillaume the PRs below were cherry-picked
> >> > I will rerun the fs and ceph-volume tests when the build is done
> >> >
> >> > https://github.com/ceph/ceph/pull/62492/commits
> >> > https://github.com/ceph/ceph/pull/62178/commits
> >> >
> >> > On Wed, Mar 26, 2025 at 2:20 PM Travis Nielsen 
> >> wrote:
> >> > >
> >> > > Oh sorry, forget my last email, thanks Laura for pointing out the
> >> obvious that this is for reef, not squid!
> >> > >
> >> > > On Wed, Mar 26, 2025 at 2:46 PM Travis Nielsen  >
> >> wrote:
> >> > >>
> >> > >> Yuri, as of when did 18.2.5 include the latest squid branch? If [1]
> >> is included in 18.2.5, then we really need [2] merged before release,
> as it
> >> would be blocking Rook.
> >> > >>
> >> > >> [1] https://github.com/ceph/ceph/pull/62095 (merged to squid on
> >> March 19)
> >> > >> [2] https://tracker.ceph.com/issues/70667
> >> > >>
> >> > >> Thanks!
> >> > >> Travis
> >> > >>
> >> > >> On Wed, Mar 26, 2025 at 2:09 PM Ilya Dryomov 
> >> wrote:
> >> > >>>
> >> > >>> On Mon, Mar 24, 2025 at 10:40 PM Yuri Weinstein <
> >> ywein...@redhat.com> wrote:
> >> > >>> >
> >> > >>> > Details of this release are summarized here:
> >> > >>> >
> >> > >>> > https://tracker.ceph.com/issues/70563#note-1
> >> > >>> > Release Notes - TBD
> >> > >>> > LRC upgrade - TBD
> >> > >>> >
> >> > >>> > Seeking approvals/reviews for:
> >> > >>> >
> >> > >>> > smoke - Laura approved?
> >> > >>> >
> >> > >>> > rados - Radek, Laura approved? Travis?  Nizamudeen? Adam King
> >> approved?
> >> > >>> >
> >> > >>> > rgw - Adam E approved?
> >> > >>> >
> >> > >>> > fs - Venky is fixing QA suite, will need to be added and rerun
> >> > >>> >
> >> > >>> > orch - Adam King approved?
> >> > >>> >
> >> > >>> > rbd - Ilya approved?
> >> > >>> > krbd - Ilya approved?
> >> > >>>
> >> > >>> Hi Yuri,
> >> > >>>
> >> > >>> rbd and krbd approved.
> >> > >>>
> >> > >>> > upgrade-clients:client-upgrade-octopus-reef-reef - Ilya please
> >> take a look.
> >> > >>>
> >> > >>> I don't recall seeing this before -- try rerunning it a couple of
> >> times?
> >> > >>>
> >> > >>> Thanks,
> >> > >>>
> >> > >>> Ilya
> >> > >>> ___
> >> > >>> ceph-users mailing list -- ceph-users@ceph.io
> >> > >>> To unsubscribe send an email to ceph-users-le...@ceph.io
> >> ___
> >> Dev mailing list -- d...@ceph.io
> >> To unsubscribe send an email to dev-le...@ceph.io
> >>
> >
> >
> > --
> >
> > Laura Flores
> >
> > She/Her/Hers
> >
> > Software Engineer, Ceph Storage 
> >
> > Chicago, IL
> >
> > lflo...@ibm.com | lflo...@redhat.com 
> > M: +17087388804
> >
> >
> >
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>


-- 

Nizamudeen A

Sr. Software Engineer - IBM

Partner Engineer


IBM and Red Hat Ceph Storage

Red Hat 

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] endless remapping after increasing number of PG in a pool

[Michel Jouvin]

Hi,

We are observing a new strange behaviour on our production cluster : we 
increased the number of PG (from 256 to 2048) in a (EC) pool after a 
warning that there was a very high number of objects per pool (the pool 
has 52M objects).


Background: this happens in the cluster that had a strange problem last 
week, discussed in the thread "Production cluster in bad shape after 
several OSD crashes". The PG increase was done after the cluster 
returned to a normal state.


The increase of the number of PG resulted in 20% misplaced objects and 
~160 PG remapped (over 256). As there is no much user activity on this 
cluster (except on this pool) these days, we decided to set the mclock 
profile to high_recovery_ops. We also disabled autoscaler on this pool 
(it was enabled and it is not clear why we add the warning with 
autoscaler enabled). The pool was created with --bulk.


The remapping went steadily during 2-3 days (as much as we can tell) but 
when reaching the end (between 0.5% and 1% of misplaced objects, ~10 PG 
remapped) it readds remapped PG (that can be seen with 'ceph pg 
dump_stuck'), all belonging to the pool affected by the increase. This 
already happened 3-4 times and it is very unclear why. There was no 
specific problem reported on the cluster that may explain this (no OSD 
down). I was wondering if the balancer may be responsible for this but I 
don't have the feeling it is the case: first the balancer doesn't report 
about doing something (but I may miss the history), second the balancer 
would probably affect PG from different pools. (there are 50 ppols in 
the cluster). There are 2 warnings that may or may not be related:


- All mon have their data size > 15 GB (mon_data_size_warn). Currently 
16 GB. It happened progressively during the remapping on all 3 mon, I 
guess it is due to the operation in progress and is harmless. Do you 
confirm?


- Since yesterday we have 4 PG that have not been deep scrubbed in time, 
belonging to different pools. Again, I tend to correlate this to the 
remapping in progress putting too much load (or other constraints), as 
there are a lot of deep scrubs per day. The current age distribution of 
deep scrubs is:


  4 "2025-03-19
 21 "2025-03-20
 46 "2025-03-21
 35 "2025-03-22
 81 "2025-03-23
    597 "2025-03-24
   1446 "2025-03-25
   2234 "2025-03-26
   2256 "2025-03-27
   1625 "2025-03-28
   1980 "2025-03-29
   2993 "2025-03-30
   3871 "2025-03-31
   1113 "2025-04-01

Should we worry about the situation? If yes, what would you advise to 
look at or do? To clear the problem last week, we had to restart all OSD 
but we didn't restart mon. Do they play a role in deciding the remapping 
plan? Is restarting them something that may help?


As usual, thanks in advance for any help/hint.

Best regards,

Michel
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: endless remapping after increasing number of PG in a pool

[Burkhard Linke]

Hi,

On 4/1/25 10:03, Michel Jouvin wrote:

Hi Bukhard,

Thanks for your answer. Your explanation seems to match well our 
observations, in particular the fact that new misplaced objects are 
added when we fall under something like 0.5% of misplaced objects. 
What is not clear for me anyway is that 'ceph osd pool ls detail' for 
the pool modified is not reporting the new pg_num target (2048) but 
the old one (256):


pool 62 'ias-z1.rgw.buckets.data' erasure profile k9_m6_host size 15 
min_size 10 crush_rule 3 object_hash rjenkins pg_num 323 pgp_num 307 
pg_num_target 256 pgp_num_target 256 autoscale_mode off last_change 
439681 lfor 0/439680/439678 flags hashpspool,bulk max_bytes 
200 stripe_width 36864 application rgw


- Is this caused by the fact that autoscaler was still on when I 
increased the number of PG and that I disabled it on the pool ~12h 
after entering the command to extend it?


This seems to be the case. The pg(p)_target settings are the number of 
PG the pool _should_ have; pg(p)_num is the number of PGs the pool 
current has. So the cluster is not splitting PGs, but merging them. If 
you want to have 2048, you should increase it again.


There are also setting for the autoscaler, e.g. 'pg_num_min'. You can 
use them to prevent the autoscaler from switching back to 256 PGs again.




- Or was it a mistake of mine to enter only extend the pg_num and not 
the pgp_num. According to the doc that I just read again, both should 
be extended at the same time or it not causing the expected result? If 
it is the case, should I just reenter the command to extend pg_num and 
pgp_num? (and wait for the resulting remapping!)


In current ceph release only pg_num can be changed. pgp_num is 
automatically adopted.



Best regards,

Burkhard Linke

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] [grafana] ceph-cluster-advanced: wrong title for object count

[Eugen Block]

Hi Ankush,

I found another (minor) Grafana issue. The "Ceph Cluster - Advanced"  
panel contains a graph for "object count", but its title is "OSD Type  
Count". I see that in Squid 19.2.0 (Grafana version 9.4.12) and 19.2.1  
(Grafana 10.4.0), and the latest ceph-cluster-advanced.json [0] also  
still contains that mistake.


Regards,
Eugen

[0]  
https://github.com/ceph/ceph/blob/89f21db321ddd404654b0064a61b0aa5428d6f7e/monitoring/ceph-mixin/dashboards_out/ceph-cluster-advanced.json#L2235

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: reef 18.2.5 QE validation status

[Adam Emerson]

On 31/03/2025, Yuri Weinstein wrote:
> Release Notes -
> 
> https://github.com/ceph/ceph/pull/62589
> https://github.com/ceph/ceph.io/pull/856

[snip]
> rgw - Adam E approved?


RGW approved!
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Prometheus anomaly in Reef

[Eugen Block]

Hi Tim,

I'm glad you sorted it out. But I'm wondering, did the prometheus spec  
file ever work? I had just assumed that it had since you wrote you had  
prometheus up and running before, so I didn't even question the  
"networks" parameter in there. Now that you say you only used the  
"--placement" parameter, I started to wonder. Since it's the only  
"extra" paramter in your spec file, I must assume that it's the root  
cause of the failures. That would probably easy to test... if you  
still have the patience. ;-)


Zitat von Tim Holloway :


Let me close this out by describing where I went afterwards.

I did a ceph orch apply with explicit --placement of 2 servers, one  
being the existing ceph08 and the other being the dell02 that was  
getting the "unknown service" errors.


This caused the dell02 machine to get a working prometheus and shut  
up about the error. The ceph08 machine reported an error. It claimed  
that apparently the prometheus port was already in use. As it was,  
since prometheus was already running. The deployer should properly  
have understood that and acted in a way that allowed the deployment  
to process without complaint.


To clear the port in use issue, I restarted prometheus on ceph08,  
then when it persisted did a "ceph mgr fail". That cleared all of  
the prometheus-related complaints and gave me a "HEALTH OK" status.


I wasn't brave enough to attempt the YAML version again, considering  
that's where the problem started. I also didn't attempt to try an  
"orch apply" that omitted any running service host, for fear it  
wouldn't remove the omitted host.


So the problem is fixed, but it took a lot of banging and hammering  
to make it work.


 Tim


On 3/28/25 19:12, Tim Holloway wrote:
Almost forgot to say. I switched out disks and got rid of the OSD  
errors. I actually found a third independent location, so it should  
be a lot more failure resistant now.


So now it's only the prometheus stuff that's still complaining.  
Everything else is happy.

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io



___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Updating ceph to pacific and quince

[Eugen Block]

Hi,

first of all, the Ceph docs are "official", here's the relevant  
section for upgrading Ceph:


https://docs.ceph.com/en/latest/cephadm/upgrade/

Octopus was the first version using the orchestrator, not Pacific. So  
you could adopt your cluster to cephadm in your current version already:


https://docs.ceph.com/en/octopus/cephadm/adoption/

There are many people on this list (and probably off list, too) who  
don't really like the orchestrator, or don't like containers etc. It  
will require some time to get used to it, but I find the benefits  
worth it. One of them being able to upgrade your cluster without  
touching the host OS. I would recommend to get some practice in a lab  
cluster and familiarize yourself with cephadm before practicing on a  
production environment. ;-)


One of the questions that arise is whether the clients (depending on  
the OpenStack version) in Octopus and the mons, mrgs, osds, mds etc.  
etc. in Pacific/Quincy will function correctly.


Note that Quincy is already EOL, you could go from Pacific to Reef  
directly (18.2.5 will be released soon). You could also go from  
Octopus to Quincy and then Squid, all those upgrade paths are safe and  
supported. Your OpenStack clients should also work fine with a newer  
Ceph cluster, just recently I upgraded a customer Ceph cluster to Reef  
while their OpenStack clients are still on Octopus, and there haven't  
been any complaints yet.


Hope this helps!
Eugen

Zitat von Iban Cabrillo :


Dear cephers,


We intend to begin the migration of our Ceph cluster from Octopus to  
Pacific and subsequently to Quincy. I have seen that from Pacific  
onwards, it is possible to automate installations with cephadm.





One of the questions that arise is whether the clients (depending on  
the OpenStack version) in Octopus and the mons, mrgs, osds, mds etc.  
etc. in Pacific/Quincy will function correctly.





The second is whether it is feasible, advisable to switch to  
cephadm/orch, since I have always performed updates manually for  
many, many years.





And the third, if there is any 'official' guide available for these updates.
Thanks in advance, I


--


Ibán Cabrillo Bartolomé
Instituto de Física de Cantabria (IFCA-CSIC)
Santander, Spain
Tel: +34942200969/+34669930421
Responsible for advanced computing service (RSC)
=
=
All our suppliers must know and accept IFCA policy available at:

https://confluence.ifca.es/display/IC/Information+Security+Policy+for+External+Suppliers
==

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io



___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: ceph.log using seconds since epoch instead of date/time stamp

[Dan van der Ster]

Hi Eugen & Bryan,

I've been trying to understand this issue -- I can't find anything in
17.2.8 that fixed it.
Should we just assume that by switching the base image to el9 from el8 fixed it?
Do we need to create a ticket here to look into this further?

Cheers, Dan

On Fri, Feb 21, 2025 at 9:44 AM Eugen Block  wrote:
>
> I looked at that last year, I'm sure this is fixed in 17.2.8 and newer.
>
> Zitat von "Stillwell, Bryan" :
>
> > Does anyone know why log entries in ceph.log and ceph.audit.log
> > would use seconds since epoch on Quincy 17.2.7/Ubuntu 20.04, but
> > ceph-mon.*.log and ceph-osd.*.log use date/time stamps?
> >
> > Example:
> > # ceph.log
> > 1740114003.665984 osd.243 (osd.243) 2137 : cluster [DBG] 3.1615 scrub ok
> >
> > # ceph-mon.log
> > 2025-02-21T00:00:01.814-0500 7fe33378e700  0 mon.XX@0(leader) e2
> > handle_command mon_command({"prefix": "osd dump"} v 0) v1
> >
> > This doesn't seem to be the case on my home cluster running Squid
> > 19.2.1/Ubuntu 24.04.
> >
> > I found this Proxmox forum post that seems to imply that could be a
> > distro setting (same Ceph version, but bookworm is affected, and
> > bullseye is not):
> >
> > https://forum.proxmox.com/threads/ceph-log-file-switched-to-unix-timestamp.138078/
> >
> > Thanks,
> > Bryan
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
>
>
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io

-- 
Dan van der Ster
Ceph Executive Council | CTO @ CLYSO
Try our Ceph Analyzer -- https://analyzer.clyso.com/
https://clyso.com | dan.vanders...@clyso.com
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Updating ceph to pacific and quince

[Iban Cabrillo]

Dear cephers, 


We intend to begin the migration of our Ceph cluster from Octopus to Pacific 
and subsequently to Quincy. I have seen that from Pacific onwards, it is 
possible to automate installations with cephadm. 




One of the questions that arise is whether the clients (depending on the 
OpenStack version) in Octopus and the mons, mrgs, osds, mds etc. etc. in 
Pacific/Quincy will function correctly. 




The second is whether it is feasible, advisable to switch to cephadm/orch, 
since I have always performed updates manually for many, many years. 




And the third, if there is any 'official' guide available for these updates. 
Thanks in advance, I 


-- 

 
Ibán Cabrillo Bartolomé 
Instituto de Física de Cantabria (IFCA-CSIC) 
Santander, Spain 
Tel: +34942200969/+34669930421 
Responsible for advanced computing service (RSC) 
=
 
=
 
All our suppliers must know and accept IFCA policy available at: 

https://confluence.ifca.es/display/IC/Information+Security+Policy+for+External+Suppliers
 
==
 

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: reef 18.2.5 QE validation status

[Yuri Weinstein]

Thank you all for review and approval

The only remaining issues are:

upgrade-clients:client-upgrade-octopus-reef-reef - Ilya and Josh
looked with no conclusion yet
upgrade/pacific-x (reef) - Laura is looking

Neha, can you take a look at those and reply with recommendations or
approval as is?

On Tue, Apr 1, 2025 at 8:32 AM Adam Emerson  wrote:
>
> On 31/03/2025, Yuri Weinstein wrote:
> > Release Notes -
> >
> > https://github.com/ceph/ceph/pull/62589
> > https://github.com/ceph/ceph.io/pull/856
>
> [snip]
> > rgw - Adam E approved?
>
>
> RGW approved!
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Updating ceph to pacific and quince

[Anthony D'Atri]

This, gentle readers, is why the Ceph community is without equal.

> On Apr 1, 2025, at 6:44 PM, Tim Holloway  wrote:
> 
> but I think we've reached the point where if you ask on the list, we can 
> clear that.

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: reshard stale-instances

[Richard Bade]

Thanks for that Casey.
The docs are a bit sparse on these commands and it just told me
"ERROR: bucket name not specified" when I ran it without anything.
After a bit of googling I was able to find a mailing list response
with this information from J. Eric Ivancich from a couple of years
ago:

> When the admin runs “bi purge” they have the option of supplying a bucket_id 
> with the “--bucket-id” command-line argument. This was useful back when 
> resharding did not automatically remove the older bucket index shards (which 
> it now does), which had a different bucket_id from the current bucket index 
> shards.

This looks to be exactly what I needed.

So I'm going to try the following using my previous example
"bstor08:2efbe9f5-7b36-4290-a370-52f57c6353c7.2708501101.618772" from
my stale instance list where bucket_name=bstor08 and
bucket_id=2efbe9f5-7b36-4290-a370-52f57c6353c7.2708501101.618772
- radosgw-admin bi purge --bucket-id="{bucket_id}"
- radosgw-admin metadata rm bucket.instance:{bucket_name}:{bucket_id}

That seems to cover what the code is doing. So hopefully this gets me
all cleaned up.

Thanks,
Rich

On Tue, 1 Apr 2025 at 02:16, Casey Bodley  wrote:
>
> On Sun, Mar 30, 2025 at 7:21 PM Richard Bade  wrote:
> >
> > Hi Everyone,
> > I've found through a bit of digging that if I use the reshard
> > stale-instances list I get entries of this form (bucket name is
> > "bstor08")
> > bstor08:2efbe9f5-7b36-4290-a370-52f57c6353c7.2708501101.618772
> > I know that this particular bucket is junk left over from an automated
> > system as per above so I'd like to clean them up.
> > I can do a metadata get on this:
> > radosgw-admin metadata get
> > bucket.instance:bstor08:2efbe9f5-7b36-4290-a370-52f57c6353c7.2708501101.618772
> > and therefore also a metadata rm:
> > radosgw-admin metadata rm
> > bucket.instance:bstor08:2efbe9f5-7b36-4290-a370-52f57c6353c7.2708501101.618772
> > After doing this I re-run the stale-instances list and that instance
> > is gone from the list so that looks promising.
> >
> > Checking the code, it looks like the stale-instance delete/rm also
> > does an instance purge before the metadata remove, so above is
> > possibly half right:
> >
> >int ret = bucket->purge_instance(dpp, y);
> >if (ret == 0){
> >  auto md_key = "bucket.instance:" +
> > binfo.bucket.get_key();
> >  ret = driver->meta_remove(dpp, md_key, y);
> >}
> >
> > Is anyone able to tell me if there's a radosgw-admin command to do the
> > instance purge part? I couldn't see anything when I was digging around
> > but I've probably missed something obvious.
>
> 'radosgw-admin bi purge' is probably what you're looking for. it
> deletes the bucket's index shard objects, but needs that bucket
> instance metadata to know how many shards there are and where they
> live
>
> >
> > Thanks,
> > Rich
> >
> > On Mon, 17 Mar 2025 at 17:49, Richard Bade  wrote:
> > >
> > > Hi Everyone,
> > > We've just completed undoing multisite to bring us back down to a
> > > single zone/site. Now I'm working through cleaning things up. Heads
> > > up, I'll probably have a few more questions over the next couple of
> > > weeks :)
> > > Firstly though, regarding "radosgw-admin reshard stale-instances
> > > list". This brings back a decent list mostly of some test buckets that
> > > were being deleted and recreated by an automated system. This has left
> > > over 150k stale instances for each of three buckets. I know these ones
> > > will be fine to delete as even if it messes up the bucket I can delete
> > > and re-create them.
> > > So my question is, how do I clean these up without using the big
> > > hammer "radosgw-admin reshard stale-instances delete" to remove all of
> > > them which would include some production buckets that I'm not quite
> > > sure how to verify are stale?
> > > Another thought that I had is maybe if I were to reshard all the
> > > buckets up to the next prime now that we're on single zone would that
> > > guarantee that all the stale instances are actually stale?
> > > Any thoughts and tips would be much appreciated.
> > >
> > > Thanks,
> > > Rich
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
> >
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Updating ceph to pacific and quince

[Tim Holloway]

As Eugen has noted, cephadm/containers were already available in 
Octopus. In fact, thanks to the somewhat scrambled nature of the 
documents, I had a mix of both containerized and legacy OSDs under 
Octopus and for the most part had no issues attributable to that.


My bigger problem with Octopus was that the deployment scheduler would 
hang if the system wasn't totally healthy. Pacific is much more 
tolerant, so life got a lot easier when I moved up.


I am a major fan of containers. I have most of my production services 
in-house containerized at this point. I never really felt comfortable 
with dumping a heterogeneous mish-mash of critical apps on a single 
machine even with the package manager riding herd on them. VMs were 
better, but hungry. Containers are very lightweight and many of the Ceph 
containers are running different service off a common base image which 
makes them even lighter.


The biggest argument I've heard against containers is security, but I 
don't think that Ceph requires any of the elevated security options that 
have proven problematic. The real reason why I think many resist 
containers is that they don't understand how they work. They are a 
different sort of world, but not that complicated really. Especially 
since Ceph's internal administration handles most of it.


One difference you will see between administered (containerised) and 
legacy Ceph resources is that the administered stuff supports more than 
one FSID per host. The downside of that is that instead of saying 
"sysadmin restart osd.4", you have to include the entire fsid in your 
control commands. For example "sysadmin restart 
ceph-2993cd86-0551-01ee-aadf-8bc5c3286cf8f-osd-4". This also shows up in 
the /var/lib/ceph directory, which now has a subdirectory for each fsid.


There is a simple way to convert a legacy ceph OSD to an administered 
one (it's in the docs). Sometime the process with jam, especially if 
things aren't as clean as they ought to be, but I think we've reached 
the point where if you ask on the list, we can clear that.


  Hope that helps,

 Tim

On 4/1/25 16:40, Eugen Block wrote:

Hi,

first of all, the Ceph docs are "official", here's the relevant 
section for upgrading Ceph:


https://docs.ceph.com/en/latest/cephadm/upgrade/

Octopus was the first version using the orchestrator, not Pacific. So 
you could adopt your cluster to cephadm in your current version already:


https://docs.ceph.com/en/octopus/cephadm/adoption/

There are many people on this list (and probably off list, too) who 
don't really like the orchestrator, or don't like containers etc. It 
will require some time to get used to it, but I find the benefits 
worth it. One of them being able to upgrade your cluster without 
touching the host OS. I would recommend to get some practice in a lab 
cluster and familiarize yourself with cephadm before practicing on a 
production environment. ;-)


One of the questions that arise is whether the clients (depending on 
the OpenStack version) in Octopus and the mons, mrgs, osds, mds etc. 
etc. in Pacific/Quincy will function correctly.


Note that Quincy is already EOL, you could go from Pacific to Reef 
directly (18.2.5 will be released soon). You could also go from 
Octopus to Quincy and then Squid, all those upgrade paths are safe and 
supported. Your OpenStack clients should also work fine with a newer 
Ceph cluster, just recently I upgraded a customer Ceph cluster to Reef 
while their OpenStack clients are still on Octopus, and there haven't 
been any complaints yet.


Hope this helps!
Eugen

Zitat von Iban Cabrillo :


Dear cephers,


We intend to begin the migration of our Ceph cluster from Octopus to 
Pacific and subsequently to Quincy. I have seen that from Pacific 
onwards, it is possible to automate installations with cephadm.





One of the questions that arise is whether the clients (depending on 
the OpenStack version) in Octopus and the mons, mrgs, osds, mds etc. 
etc. in Pacific/Quincy will function correctly.





The second is whether it is feasible, advisable to switch to 
cephadm/orch, since I have always performed updates manually for 
many, many years.





And the third, if there is any 'official' guide available for these 
updates.

Thanks in advance, I


--


Ibán Cabrillo Bartolomé
Instituto de Física de Cantabria (IFCA-CSIC)
Santander, Spain
Tel: +34942200969/+34669930421
Responsible for advanced computing service (RSC)
= 

= 


All our suppliers must know and accept IFCA policy available at:

https://confluence.ifca.es/display/IC/Information+Security+Policy+for+External+Suppliers 

== 



__

[ceph-users] Re: Major version upgrades with CephADM

[Dominique Ramaekers]

Hi Alex,

My own sysadmin experience...
Did a major version upgrade two times now always to a version higher in minor 
versioning than the '.0'-version. Without any issues regarding skipping the 
'.0'-version.

But also upgrading to for instance the v18 image. I never specify the minor 
versioning (so not v18.2.1). This way, the upgrade is done to the minor version 
which is advised. You can trace which major version correspond to which minor 
here: https://quay.io/repository/ceph/ceph?tab=tags

Ex. v18 correspond to v18.2.4-20240724 not v18.2.2...

Greetings,

Dominique.

> -Oorspronkelijk bericht-
> Van: Tim Holloway 
> Verzonden: dinsdag 1 april 2025 17:45
> Aan: ceph-users@ceph.io
> Onderwerp: [ceph-users] Re: Major version upgrades with CephADM
> 
> Well, I just upgraded Pacific to Reef 18.2.4 and the only problems I ran into
> had been problems previously seen in Pacific.
> 
> Your Mileage May Vary. as the only part of Ceph I put a strain on is deploying
> stuff and adding/removing OSDs, but for the rest, I'd look to recent Reef
> complaints on the list and see if any of them apply.
> 
>   Regards,
> Tim
> 
> On Mon, 2025-03-31 at 21:37 -0400, Alex Petty wrote:
> > Hello,
> >
> > I'm seeking guidance on best practices for major version upgrades with
> > Cephadm managed clusters. I have a production Pacfic 16.2.15 cluster
> > that has had minor version upgrades with Cephadm, but always just a
> > single step at a time. We plan to upgrade to Reef soon. Is it best to
> > go straight to the most recent Reef release available, which is 18.2.4
> > at this time?
> > Or
> > could there be reasons to start at 18.2.0?
> >
> > Is the best source of documentation about major version upgrades the
> > blog posts when a major release happens:
> > https://ceph.io/en/news/blog/2023/v18-2-0-reef-released/#upgrade ?
> >
> > Thanks,
> >
> > Alex
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an
> > email to ceph-users-le...@ceph.io
> ___
> ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email
> to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Major version upgrades with CephADM

[Tim Holloway]

Well, I just upgraded Pacific to Reef 18.2.4 and the only problems I
ran into had been problems previously seen in Pacific.

Your Mileage May Vary. as the only part of Ceph I put a strain on is
deploying stuff and adding/removing OSDs, but for the rest, I'd look to
recent Reef complaints on the list and see if any of them apply.

  Regards,
Tim

On Mon, 2025-03-31 at 21:37 -0400, Alex Petty wrote:
> Hello,
> 
> I'm seeking guidance on best practices for major version upgrades
> with
> Cephadm managed clusters. I have a production Pacfic 16.2.15 cluster
> that
> has had minor version upgrades with Cephadm, but always just a single
> step
> at a time. We plan to upgrade to Reef soon. Is it best to go straight
> to
> the most recent Reef release available, which is 18.2.4 at this time?
> Or
> could there be reasons to start at 18.2.0?
> 
> Is the best source of documentation about major version upgrades the
> blog
> posts when a major release happens:
> https://ceph.io/en/news/blog/2023/v18-2-0-reef-released/#upgrade ?
> 
> Thanks,
> 
> Alex
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: ceph.log using seconds since epoch instead of date/time stamp

[Eugen Block]

Hi Dan,

I just found my notes from last year, unfortunately without any  
"evidence". So I checked one of my lab clusters running with 17.2.8  
(so it's el9), but the ceph.log is still using the epoch timestamp:


quincy-1:~ # tail /var/log/ceph/1e6e5cb6-73e8-11ee-b195-fa163ee43e22/ceph.log
1743524881.6342404 mgr.quincy-1.pcasdd (mgr.2614124) 65 : cluster  
[DBG] pgmap v58: 361 pgs: 361 active+clean; 4.9 GiB data, 17 GiB used,  
53 GiB / 70 GiB avail; 102 B/s rd, 0 op/s


Note that I have log_to_file enabled, not sure how much a difference  
that makes. And I don't recall what my assumption was based on, but  
apparently it's not correct. I thought I had found a tracker last  
year, pointing to a pull request, but as I said, I didn't write down  
much. I'll keep looking if I find anything else.


Zitat von Dan van der Ster :


Hi Eugen & Bryan,

I've been trying to understand this issue -- I can't find anything in
17.2.8 that fixed it.
Should we just assume that by switching the base image to el9 from  
el8 fixed it?

Do we need to create a ticket here to look into this further?

Cheers, Dan

On Fri, Feb 21, 2025 at 9:44 AM Eugen Block  wrote:


I looked at that last year, I'm sure this is fixed in 17.2.8 and newer.

Zitat von "Stillwell, Bryan" :

> Does anyone know why log entries in ceph.log and ceph.audit.log
> would use seconds since epoch on Quincy 17.2.7/Ubuntu 20.04, but
> ceph-mon.*.log and ceph-osd.*.log use date/time stamps?
>
> Example:
> # ceph.log
> 1740114003.665984 osd.243 (osd.243) 2137 : cluster [DBG] 3.1615 scrub ok
>
> # ceph-mon.log
> 2025-02-21T00:00:01.814-0500 7fe33378e700  0 mon.XX@0(leader) e2
> handle_command mon_command({"prefix": "osd dump"} v 0) v1
>
> This doesn't seem to be the case on my home cluster running Squid
> 19.2.1/Ubuntu 24.04.
>
> I found this Proxmox forum post that seems to imply that could be a
> distro setting (same Ceph version, but bookworm is affected, and
> bullseye is not):
>
>  
https://forum.proxmox.com/threads/ceph-log-file-switched-to-unix-timestamp.138078/

>
> Thanks,
> Bryan
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io


___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io


--
Dan van der Ster
Ceph Executive Council | CTO @ CLYSO
Try our Ceph Analyzer -- https://analyzer.clyso.com/
https://clyso.com | dan.vanders...@clyso.com



___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: endless remapping after increasing number of PG in a pool

[Burkhard Linke]

Hi,

On 4/1/25 09:06, Michel Jouvin wrote:

Hi,

We are observing a new strange behaviour on our production cluster : 
we increased the number of PG (from 256 to 2048) in a (EC) pool after 
a warning that there was a very high number of objects per pool (the 
pool has 52M objects).


Background: this happens in the cluster that had a strange problem 
last week, discussed in the thread "Production cluster in bad shape 
after several OSD crashes". The PG increase was done after the cluster 
returned to a normal state.


The increase of the number of PG resulted in 20% misplaced objects and 
~160 PG remapped (over 256). As there is no much user activity on this 
cluster (except on this pool) these days, we decided to set the mclock 
profile to high_recovery_ops. We also disabled autoscaler on this pool 
(it was enabled and it is not clear why we add the warning with 
autoscaler enabled). The pool was created with --bulk.


The remapping went steadily during 2-3 days (as much as we can tell) 
but when reaching the end (between 0.5% and 1% of misplaced objects, 
~10 PG remapped) it readds remapped PG (that can be seen with 'ceph pg 
dump_stuck'), all belonging to the pool affected by the increase. This 
already happened 3-4 times and it is very unclear why. There was no 
specific problem reported on the cluster that may explain this (no OSD 
down). I was wondering if the balancer may be responsible for this but 
I don't have the feeling it is the case: first the balancer doesn't 
report about doing something (but I may miss the history), second the 
balancer would probably affect PG from different pools. (there are 50 
ppols in the cluster). There are 2 warnings that may or may not be 
related:


*snipsnap*

you are probably seeing the loadbalancer in action. If you increase the 
number of PGs, this change is performed gradually. You can check this by 
having a look at the output of 'ceph osd pool ls detail'. It will print 
the current number of PGs _and_ PGPs, and the target number of both 
values. If you increase the number of PG, existing PGs have to be 
splitted and a part of the data has to be transferred to the new PG. The 
loadbalancer will take care for this and start the next PG split if the 
number of misplaced objects falls under a certain threshold. The default 
value is 0.5% is I remember correctly.




- All mon have their data size > 15 GB (mon_data_size_warn). Currently 
16 GB. It happened progressively during the remapping on all 3 mon, I 
guess it is due to the operation in progress and is harmless. Do you 
confirm?
The mons have to save historic osd and pg maps as long as the cluster is 
not healthy. During large data movement operations this might pile up to 
a significant size. You should monitor the available space and ensure 
that the mons are not running out of disk space. I'm not sure whether 
manual intervention like a manual compaction will help in this case


- Since yesterday we have 4 PG that have not been deep scrubbed in 
time, belonging to different pools. Again, I tend to correlate this to 
the remapping in progress putting too much load (or other 
constraints), as there are a lot of deep scrubs per day. The current 
age distribution of deep scrubs is:


  4 "2025-03-19
 21 "2025-03-20
 46 "2025-03-21
 35 "2025-03-22
 81 "2025-03-23
    597 "2025-03-24
   1446 "2025-03-25
   2234 "2025-03-26
   2256 "2025-03-27
   1625 "2025-03-28
   1980 "2025-03-29
   2993 "2025-03-30
   3871 "2025-03-31
   1113 "2025-04-01

Should we worry about the situation? If yes, what would you advise to 
look at or do? To clear the problem last week, we had to restart all 
OSD but we didn't restart mon. Do they play a role in deciding the 
remapping plan? Is restarting them something that may help?


Remapping PGs has a higher priority than scrubbing. So as long as the 
current pool extension is not finished, only idle OSDs will scrub their 
PGs. This is expected, and the cluster will take care for the missing 
scrub runs after it is healthy again.



Best regards,

Burkhard Linke

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Major version upgrades with CephADM

[Joel Davidow]

In addition to the blog posts, as part of my planning for an upgrade, I check 
several places for potential issues and changes:
  1) https://docs.ceph.com/en//cephadm/upgrade/ 
  2) https://docs.ceph.com/en/latest/releases// 
  3) https://tracker.ceph.com/
  4) this mailing list

I also test the upgrade and write an upgrade procedure that includes 
pre-upgrade steps, emergency procedures, and post-upgrade steps, which are 
important for cephadm as the ceph command and cephadm have to be upgraded 
separately plus any scripts need to be tested and updated as needed.

I upgraded 16.2.15 directly to 18.2.4 with cephadm using a staggered upgrade 
(available since 16.2.11) and the upgrade went fine. I specified a particular 
image using --image with the ceph orch upgrade start command.

Cheers,
Joel
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: endless remapping after increasing number of PG in a pool

[Michel Jouvin]

Hi Bukhard,

Thanks for your answer. Your explanation seems to match well our 
observations, in particular the fact that new misplaced objects are 
added when we fall under something like 0.5% of misplaced objects. What 
is not clear for me anyway is that 'ceph osd pool ls detail' for the 
pool modified is not reporting the new pg_num target (2048) but the old 
one (256):


pool 62 'ias-z1.rgw.buckets.data' erasure profile k9_m6_host size 15 
min_size 10 crush_rule 3 object_hash rjenkins pg_num 323 pgp_num 307 
pg_num_target 256 pgp_num_target 256 autoscale_mode off last_change 
439681 lfor 0/439680/439678 flags hashpspool,bulk max_bytes 
200 stripe_width 36864 application rgw


- Is this caused by the fact that autoscaler was still on when I 
increased the number of PG and that I disabled it on the pool ~12h after 
entering the command to extend it?


- Or was it a mistake of mine to enter only extend the pg_num and not 
the pgp_num. According to the doc that I just read again, both should be 
extended at the same time or it not causing the expected result? If it 
is the case, should I just reenter the command to extend pg_num and 
pgp_num? (and wait for the resulting remapping!).


Best regards,

Michel


Le 01/04/2025 à 09:32, Burkhard Linke a écrit :

Hi,

On 4/1/25 09:06, Michel Jouvin wrote:

Hi,

We are observing a new strange behaviour on our production cluster : 
we increased the number of PG (from 256 to 2048) in a (EC) pool after 
a warning that there was a very high number of objects per pool (the 
pool has 52M objects).


Background: this happens in the cluster that had a strange problem 
last week, discussed in the thread "Production cluster in bad shape 
after several OSD crashes". The PG increase was done after the 
cluster returned to a normal state.


The increase of the number of PG resulted in 20% misplaced objects 
and ~160 PG remapped (over 256). As there is no much user activity on 
this cluster (except on this pool) these days, we decided to set the 
mclock profile to high_recovery_ops. We also disabled autoscaler on 
this pool (it was enabled and it is not clear why we add the warning 
with autoscaler enabled). The pool was created with --bulk.


The remapping went steadily during 2-3 days (as much as we can tell) 
but when reaching the end (between 0.5% and 1% of misplaced objects, 
~10 PG remapped) it readds remapped PG (that can be seen with 'ceph 
pg dump_stuck'), all belonging to the pool affected by the increase. 
This already happened 3-4 times and it is very unclear why. There was 
no specific problem reported on the cluster that may explain this (no 
OSD down). I was wondering if the balancer may be responsible for 
this but I don't have the feeling it is the case: first the balancer 
doesn't report about doing something (but I may miss the history), 
second the balancer would probably affect PG from different pools. 
(there are 50 ppols in the cluster). There are 2 warnings that may or 
may not be related:


*snipsnap*

you are probably seeing the loadbalancer in action. If you increase 
the number of PGs, this change is performed gradually. You can check 
this by having a look at the output of 'ceph osd pool ls detail'. It 
will print the current number of PGs _and_ PGPs, and the target number 
of both values. If you increase the number of PG, existing PGs have to 
be splitted and a part of the data has to be transferred to the new 
PG. The loadbalancer will take care for this and start the next PG 
split if the number of misplaced objects falls under a certain 
threshold. The default value is 0.5% is I remember correctly.




- All mon have their data size > 15 GB (mon_data_size_warn). 
Currently 16 GB. It happened progressively during the remapping on 
all 3 mon, I guess it is due to the operation in progress and is 
harmless. Do you confirm?
The mons have to save historic osd and pg maps as long as the cluster 
is not healthy. During large data movement operations this might pile 
up to a significant size. You should monitor the available space and 
ensure that the mons are not running out of disk space. I'm not sure 
whether manual intervention like a manual compaction will help in this 
case


- Since yesterday we have 4 PG that have not been deep scrubbed in 
time, belonging to different pools. Again, I tend to correlate this 
to the remapping in progress putting too much load (or other 
constraints), as there are a lot of deep scrubs per day. The current 
age distribution of deep scrubs is:


  4 "2025-03-19
 21 "2025-03-20
 46 "2025-03-21
 35 "2025-03-22
 81 "2025-03-23
    597 "2025-03-24
   1446 "2025-03-25
   2234 "2025-03-26
   2256 "2025-03-27
   1625 "2025-03-28
   1980 "2025-03-29
   2993 "2025-03-30
   3871 "2025-03-31
   1113 "2025-04-01

Should we worry about the situation? If yes, what would you advise to 
look at or do? To clear the problem last week, we had to restart all 
OSD but we didn't restart mon. Do they play a

[ceph-users] Re: Updating ceph to pacific and quince

[Iban Cabrillo]

Hi,
   Thanks so much, guys, for all your input and perspectives, it's been really 
enriching

Regards, I

-- 

  Ibán Cabrillo Bartolomé
  Instituto de Física de Cantabria (IFCA-CSIC)
  Santander, Spain
  Tel: +34942200969/+34669930421
  Responsible for advanced computing service (RSC)
=
=
All our suppliers must know and accept IFCA policy available at:

https://confluence.ifca.es/display/IC/Information+Security+Policy+for+External+Suppliers
==
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Purpose of the cephadm account

[dhivagar selvam]

Hi,

We have set up a ceph cluster via ceph-ansible. When we add a new osd or
mds to the ceph cluster, the "cephadm" account is automatically created
with "/bin/bash".

1. Is this account necessary?

2. Can I delete this account or do I need to change something in the
ceph-ansible configuration?


Thanks,
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: constant increase in osdmap epoch

[Joel Davidow]

I'm seeing a similar increase in osdmap epochs with the only diff from `ceph 
osd dump` being epoch and modified date. Duration of osdmap epochs varies a lot 
on scale of seconds but are generally less than 10 minutes with a max of about 
half an hour. This is in a cephadm cluster running 18.2.4 (upgraded from 
16.2.15) with rgw only. No changes to scrub configs. No snapshots, though 
`purged_snaps scrubs` are logged at debug 10.  I've set mon debug at 5, 10, 15, 
and 20 but haven't found anything in those logs that suggests a cause. There is 
a chance I've missed something in the debug 20 log though as it has 1244292 
lines. A portion of the debug 15 log is below. Any suggestions on next steps 
and/or possible causes are welcome. Thanks.

```
Mar 31 16:20:46 firelord bash[2426394]: debug 2025-03-31T16:20:46.443+ 
7f02de959640 10 mon.firelord@0(leader).osd e405676 should_propose   


 
Mar 31 16:20:46 firelord bash[2426394]: debug 2025-03-31T16:20:46.647+ 
7f02de959640 10 mon.firelord@0(leader).log v61415522 preprocess_query log(2 
entries from seq 12229 at 2025-03-31T16:20:46.448717+) v1 from osd.132 
v2:10.224.190.122:6833/3799949454   

  
Mar 31 16:20:46 firelord bash[2426394]: debug 2025-03-31T16:20:46.647+ 
7f02de959640 10 mon.firelord@0(leader).log v61415522 preprocess_log log(2 
entries from seq 12229 at 2025-03-31T16:20:46.448717+) v1 from osd.132  


   
Mar 31 16:20:46 firelord bash[2426394]: debug 2025-03-31T16:20:46.647+ 
7f02de959640 10 mon.firelord@0(leader).log v61415522 prepare_update log(2 
entries from seq 12229 at 2025-03-31T16:20:46.448717+) v1 from osd.132 
v2:10.224.190.122:6833/3799949454   


Mar 31 16:20:46 firelord bash[2426394]: debug 2025-03-31T16:20:46.647+ 
7f02de959640 10 mon.firelord@0(leader).log v61415522 prepare_log log(2 entries 
from seq 12229 at 2025-03-31T16:20:46.448717+) v1 from osd.132  


  
Mar 31 16:20:46 firelord bash[2426394]: debug 2025-03-31T16:20:46.647+ 
7f02de959640 10 mon.firelord@0(leader).log v61415522 logging 
2025-03-31T16:20:46.448717+ osd.132 (osd.132) 12229 : cluster [DBG] 
purged_snaps scrub starts   


Mar 31 16:20:46 firelord bash[2426394]: debug 2025-03-31T16:20:46.647+ 
7f02de959640 10 mon.firelord@0(leader).log v61415522 logging 
2025-03-31T16:20:46.449046+ osd.132 (osd.132) 12230 : cluster [DBG] 
purged_snaps scrub ok   


Mar 31 16:20:46 firelord bash[2426394]: debug 2025-03-31T16:20:46.723+ 
7f02de959640 10 mon.firelord@0(leader).osd e405676 preprocess_query 
osd_beacon(pgs [6.1ef,6.3d,6.2d2,6.5ee,6.102,7.85,6.6da,8.e] lec 405676 
last_purged_snaps_scrub 2025-03-30T19:54:39.335243+ 
osd_beacon_report_interval 300 v405676) v3 from osd.177 
v2:10.224.190.121:6872/1659276092   

 
Mar 31 16:20:46 firelord bash[2426394]: debug 2025-03-31T16:20:46.723+ 
7f02de959640 10 mon.firelord@0(leader) e15 no_reply to osd.177 
v2:10.224.190.121:6872/1659276092 osd_beacon(pgs 
[6.1ef,6.3d,6.2d2,6.5ee,6.102,7.85,6.6da,8.e] lec 405676 
last_purged_snaps_scrub 2025-03-30T19:54:39.335243+ 
osd_beacon_report_interval 300 v405676) v3  


Mar 31 16:20:46 mon.0 bash[2426394]: debug 2025-03-31T16:20:46.723+ 
7f02de959640 7 mon.0@0(leader).osd e405676 prepare_update osd_beacon(pgs 
[6.1ef,6.