[ceph-users] All ceph commands hangs - bad magic number in monitor log

[Evrard Van Espen - Weather-Measures]

Hello all,

We have a production cluster running since weeks deployed on centos8 
with |cephadm| (3 nodes with 6x12TB HDD each, one |mon| and one |mgr| on 
each node too). Today, the "master" node (the from on which I run all 
the setup commands) crashed so I rebooted the server. Now, all ceph 
commands hangs (|ceph -s| for ex), when I do |cephadm shell| it asks me 
to specify the FSID (showing me 3 fsid) and in the logs of the unit 
|ceph-@mon.ceph-n-01.service| I get this line looping :


|Dec 10 16:13:07 ceph-n-01 bash[36847]: debug 
2020-12-10T16:13:07.099+ 7f9251890700 0 cephx: verify_authorizer 
could not decrypt ticket info: error: bad magic in decode_decrypt, 
924161695680779075 != 18374858748799134293 Dec 10 16:13:10 ceph-n-01 
bash[36847]: debug 2020-12-10T16:13:10.288+ 7f9251890700 1 
mon.ceph-n-01@0(probing) e3 handle_auth_request failed to assign global_id |


Thanks for all your help :)

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Scrubbing - osd down

[Miroslav Boháč]

Hi,

I have a problem with crashing OSD daemons in our Ceph 15.2.6 cluster . The
problem was temporarily resolved by disabling scrub and deep-scrub. All PGs
are active+clean. After a few days I tried to enable scrubbing again, but
the problem persists. OSD with high latencies, PG laggy, osd not
responding, OSD was marked down and the cluster is not usable. Problem
appeared after failover when 10 OSD was marked as out.

I will appreciate any help or advice, how to resolve this problem.

Regards,
Miroslav





2020-12-10T21:39:57.721883+0100 osd.1 (osd.1) 5 : cluster [DBG] 18.11
deep-scrub starts

2020-12-10T21:39:57.880861+0100 osd.1 (osd.1) 6 : cluster [DBG] 18.11
deep-scrub ok

2020-12-10T21:39:58.713422+0100 osd.1 (osd.1) 7 : cluster [DBG] 43.5 scrub
starts

2020-12-10T21:39:58.719372+0100 osd.1 (osd.1) 8 : cluster [DBG] 43.5 scrub
ok

2020-12-10T21:39:59.296962+0100 mgr.pve2-prg2a (mgr.91575377) 118746 :
cluster [DBG] pgmap v119000: 1737 pgs: 3 active+clean+laggy, 3
active+clean+scrubbing+deep, 1731 active+clean; 3.3 TiB data, 7.8 TiB used,
21 TiB / 29 TiB avail; 117 KiB/s rd, 2.5 MiB/s wr, 269 op/s

2020-12-10T21:40:00.088421+0100 osd.29 (osd.29) 74 : cluster [DBG] 1.13b
deep-scrub starts

2020-12-10T21:40:01.300373+0100 mgr.pve2-prg2a (mgr.91575377) 118747 :
cluster [DBG] pgmap v119001: 1737 pgs: 3 active+clean+laggy, 3
active+clean+scrubbing+deep, 1731 active+clean; 3.3 TiB data, 7.8 TiB used,
21 TiB / 29 TiB avail; 101 KiB/s rd, 1.9 MiB/s wr, 202 op/s

2020-12-10T21:40:02.681058+0100 osd.34 (osd.34) 13 : cluster [DBG] 1.a2
deep-scrub ok

2020-12-10T21:40:03.304009+0100 mgr.pve2-prg2a (mgr.91575377) 118749 :
cluster [DBG] pgmap v119002: 1737 pgs: 3 active+clean+laggy, 3
active+clean+scrubbing+deep, 1731 active+clean; 3.3 TiB data, 7.8 TiB used,
21 TiB / 29 TiB avail; 101 KiB/s rd, 1.9 MiB/s wr, 198 op/s

2020-12-10T21:40:05.316233+0100 mgr.pve2-prg2a (mgr.91575377) 118750 :
cluster [DBG] pgmap v119003: 1737 pgs: 6 active+clean+laggy, 3
active+clean+scrubbing+deep, 1728 active+clean; 3.3 TiB data, 7.8 TiB used,
21 TiB / 29 TiB avail; 150 KiB/s rd, 3.0 MiB/s wr, 249 op/s

2020-12-10T21:40:07.319643+0100 mgr.pve2-prg2a (mgr.91575377) 118751 :
cluster [DBG] pgmap v119004: 1737 pgs: 6 active+clean+laggy, 3
active+clean+scrubbing+deep, 1728 active+clean; 3.3 TiB data, 7.8 TiB used,
21 TiB / 29 TiB avail; 142 KiB/s rd, 2.3 MiB/s wr, 212 op/s





2020-12-10T21:40:15.523134+0100 mon.pve1-prg2a (mon.0) 125943 : cluster
[DBG] osd.4 reported failed by osd.24

2020-12-10T21:40:15.523325+0100 mon.pve1-prg2a (mon.0) 125944 : cluster
[DBG] osd.39 reported failed by osd.24

2020-12-10T21:40:16.112299+0100 mon.pve1-prg2a (mon.0) 125946 : cluster
[WRN] Health check failed: 0 slow ops, oldest one blocked for 32 sec, osd.8
has slow ops (SLOW_OPS)

2020-12-10T21:40:16.202867+0100 mon.pve1-prg2a (mon.0) 125947 : cluster
[DBG] osd.4 reported failed by osd.34

2020-12-10T21:40:16.202986+0100 mon.pve1-prg2a (mon.0) 125948 : cluster
[INF] osd.4 failed (root=default,host=pve1-prg2a) (2 reporters from
different host after 24.000267 >= grace 22.361677)

2020-12-10T21:40:16.373925+0100 mon.pve1-prg2a (mon.0) 125949 : cluster
[DBG] osd.39 reported failed by osd.6

2020-12-10T21:40:16.865608+0100 mon.pve1-prg2a (mon.0) 125951 : cluster
[DBG] osd.39 reported failed by osd.8

2020-12-10T21:40:17.125917+0100 mon.pve1-prg2a (mon.0) 125952 : cluster
[WRN] Health check failed: 1 osds down (OSD_DOWN)

2020-12-10T21:40:17.139006+0100 mon.pve1-prg2a (mon.0) 125953 : cluster
[DBG] osdmap e12819: 40 total, 39 up, 30 in

2020-12-10T21:40:17.140248+0100 mon.pve1-prg2a (mon.0) 125954 : cluster
[DBG] osd.39 reported failed by osd.21

2020-12-10T21:40:17.344244+0100 mgr.pve2-prg2a (mgr.91575377) 118757 :
cluster [DBG] pgmap v119012: 1737 pgs: 9 peering, 61 stale+active+clean, 1
active+clean+scrubbing+deep, 7 active+clean+laggy, 1659 active+clean; 3.3
TiB data, 7.8 TiB used, 14 TiB / 22 TiB avail; 44 KiB/s rd, 2.5 MiB/s wr,
107 op/s

2020-12-10T21:40:17.378069+0100 mon.pve1-prg2a (mon.0) 125955 : cluster
[DBG] osd.39 reported failed by osd.26

2020-12-10T21:40:17.424429+0100 mon.pve1-prg2a (mon.0) 125956 : cluster
[DBG] osd.39 reported failed by osd.18

2020-12-10T21:40:17.829447+0100 mon.pve1-prg2a (mon.0) 125957 : cluster
[DBG] osd.39 reported failed by osd.36

2020-12-10T21:40:17.847373+0100 mon.pve1-prg2a (mon.0) 125958 : cluster
[DBG] osd.39 reported failed by osd.1

2020-12-10T21:40:17.858371+0100 mon.pve1-prg2a (mon.0) 125959 : cluster
[DBG] osd.39 reported failed by osd.17

2020-12-10T21:40:17.915755+0100 mon.pve1-prg2a (mon.0) 125960 : cluster
[DBG] osd.39 reported failed by osd.28





2020-12-10T21:40:24.151192+0100 mon.pve1-prg2a (mon.0) 125986 : cluster
[INF] Health check cleared: PG_AVAILABILITY (was: Reduced data
availability: 1 pg peering)

2020-12-10T21:40:24.608038+0100 mon.pve1-prg2a (mon.0) 125987 : cluster
[WRN] Health check update: 0 slow ops, oldest one blocked for 37 sec, osd.8
has slow ops (SLOW_OPS)

2020-12-10T21:40:25.37

[ceph-users] CephFS max_file_size

[Mark Schouten]

Hi,

There is a default limit of 1TiB for the max_file_size in CephFS. I altered 
that to 2TiB, but I now got a request for storing a file up to 7TiB.

I'd expect the limit to be there for a reason, but what is the risk of setting 
that value to say 10TiB?

--
Mark Schouten 

Tuxis, Ede, https://www.tuxis.nl

T: +31 318 200208 
 

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Larger number of OSDs, cheroot, cherrypy, limits + containers == broken

[David Orman]

Hi Ken,

This seems to have fixed that issue. It exposed another:
https://tracker.ceph.com/issues/39264 which is causing ceph-mgr to become
entirely unresponsive across the cluster, but cheroot seems to be ok.

David

On Wed, Dec 9, 2020 at 12:25 PM David Orman  wrote:

> Ken,
>
> We have rebuilt the container images of 15.2.7 with this RPM applied, and
> will be deploying it to a larger (504 OSD) cluster to test - this cluster
> had the issue previously until we disabled polling via Prometheus. We will
> update as soon as it's run for a day or two and we've been able to verify
> the mgr issues we saw no longer occur after extended polling via external
> and internal prometheus instances.
>
> Thank you again for the quick update, we'll let you know as soon as we
> have more feedback,
> David
>
> On Tue, Dec 8, 2020 at 10:37 AM David Orman  wrote:
>
>> Hi Ken,
>>
>> Thank you for the update! As per:
>> https://github.com/ceph/ceph-container/issues/1748
>>
>> We implemented the (dropping ulimit to 1024:4096 for mgr) suggested
>> change last night, and on our test cluster of 504 OSDs, being polled by the
>> internal prometheus and our external instance, the mgrs stopped responding
>> and dropped out of the cluster entirely. This is impacting not just
>> metrics, but the mgr itself. I think this is a high priority issue, as
>> metrics are critical for prod, but mgr itself seems to be impacted on a
>> moderately sized cluster.
>>
>> Respectfully,
>> David Orman
>>
>> On Mon, Dec 7, 2020 at 1:50 PM Ken Dreyer  wrote:
>>
>>> Thanks for bringing this up.
>>>
>>> We need to update Cheroot in Fedora and EPEL 8. I've opened
>>> https://src.fedoraproject.org/rpms/python-cheroot/pull-request/3 to
>>> get this into Fedora first.
>>>
>>> I've published an el8 RPM at
>>> https://fedorapeople.org/~ktdreyer/bz1868629/ for early testing. I can
>>> bring up a "hello world" cherrypy app with this, but I've not tested
>>> it with Ceph.
>>>
>>> - Ken
>>>
>>> On Mon, Dec 7, 2020 at 9:57 AM David Orman  wrote:
>>> >
>>> > Hi,
>>> >
>>> > We have a ceph 15.2.7 deployment using cephadm under podman w/ systemd.
>>> > We've run into what we believe is:
>>> >
>>> > https://github.com/ceph/ceph-container/issues/1748
>>> > https://tracker.ceph.com/issues/47875
>>> >
>>> > In our case, eventually the mgr container stops emitting
>>> output/logging. We
>>> > are polling with external prometheus clusters, which is likely what
>>> > triggers the issue, as it appears some amount of time after the
>>> container
>>> > is spawned.
>>> >
>>> > Unfortunately, setting limits in the systemd service file for the mgr
>>> > service on the host OS doesn't work, nor does modifying the unit.run
>>> file
>>> > which is used to start the container under podman to include the
>>> --ulimit
>>> > settings as suggested. Looking inside the container:
>>> >
>>> > lib/systemd/system/ceph-mgr@.service:LimitNOFILE=1048576
>>> >
>>> > This prevents us from deploying medium to large ceph clusters, so I
>>> would
>>> > argue it's a high priority bug that should not be closed, unless there
>>> is a
>>> > workaround that works until EPEL 8 contains the fixed version of
>>> cheroot
>>> > and the ceph containers include it.
>>> >
>>> > My understanding is this was fixed in cheroot 8.4.0:
>>> >
>>> > https://github.com/cherrypy/cheroot/issues/249
>>> > https://github.com/cherrypy/cheroot/pull/301
>>> >
>>> > Thank you in advance for any suggestions,
>>> > David
>>> > ___
>>> > ceph-users mailing list -- ceph-users@ceph.io
>>> > To unsubscribe send an email to ceph-users-le...@ceph.io
>>> >
>>>
>>>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Ceph benchmark tool (cbt)

[Marc Roos]

Just run the tool from a client that is not part of the ceph nodes. Than 
it can do nothing, that you did not configure ceph to allow it to do ;) 
Besides you should never run software from 'unknown' sources in an 
environment where it can use 'admin' rights.
 


-Original Message-
To: ceph-users
Subject: [ceph-users] Ceph benchmark tool (cbt)

Hi all,

I want to benchmark my production cluster with cbt. I read a bit of the 
code and I see something strange in it, for example, it's going to 
create ceph-osd by it selves (
https://github.com/ceph/cbt/blob/master/cluster/ceph.py#L373) and also 
shutdown the whole cluster!! (
https://github.com/ceph/cbt/blob/master/cluster/ceph.py#L212)

Is there any configuration to not do harmful things with the cluster and 
for example just test the read_ahead_kb or simply stop some OSDs and 
other things that can be reverted and not get the cluster fully down?!

Thanks.
___
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an 
email to ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Scrubbing - osd down

[Igor Fedotov]

Hi Miroslav,

haven't you performed massive data removal (PG migration) recently?

If so you might want to apply manual DB compaction to your OSDs.

The positive effect might be just temporary if background removals are 
still in progress though..



See https://tracker.ceph.com/issues/47044 and the references for more 
info on the issue.



Thanks,

Igor




On 12/11/2020 12:50 PM, Miroslav Boháč wrote:

Hi,

I have a problem with crashing OSD daemons in our Ceph 15.2.6 cluster . The
problem was temporarily resolved by disabling scrub and deep-scrub. All PGs
are active+clean. After a few days I tried to enable scrubbing again, but
the problem persists. OSD with high latencies, PG laggy, osd not
responding, OSD was marked down and the cluster is not usable. Problem
appeared after failover when 10 OSD was marked as out.

I will appreciate any help or advice, how to resolve this problem.

Regards,
Miroslav





2020-12-10T21:39:57.721883+0100 osd.1 (osd.1) 5 : cluster [DBG] 18.11
deep-scrub starts

2020-12-10T21:39:57.880861+0100 osd.1 (osd.1) 6 : cluster [DBG] 18.11
deep-scrub ok

2020-12-10T21:39:58.713422+0100 osd.1 (osd.1) 7 : cluster [DBG] 43.5 scrub
starts

2020-12-10T21:39:58.719372+0100 osd.1 (osd.1) 8 : cluster [DBG] 43.5 scrub
ok

2020-12-10T21:39:59.296962+0100 mgr.pve2-prg2a (mgr.91575377) 118746 :
cluster [DBG] pgmap v119000: 1737 pgs: 3 active+clean+laggy, 3
active+clean+scrubbing+deep, 1731 active+clean; 3.3 TiB data, 7.8 TiB used,
21 TiB / 29 TiB avail; 117 KiB/s rd, 2.5 MiB/s wr, 269 op/s

2020-12-10T21:40:00.088421+0100 osd.29 (osd.29) 74 : cluster [DBG] 1.13b
deep-scrub starts

2020-12-10T21:40:01.300373+0100 mgr.pve2-prg2a (mgr.91575377) 118747 :
cluster [DBG] pgmap v119001: 1737 pgs: 3 active+clean+laggy, 3
active+clean+scrubbing+deep, 1731 active+clean; 3.3 TiB data, 7.8 TiB used,
21 TiB / 29 TiB avail; 101 KiB/s rd, 1.9 MiB/s wr, 202 op/s

2020-12-10T21:40:02.681058+0100 osd.34 (osd.34) 13 : cluster [DBG] 1.a2
deep-scrub ok

2020-12-10T21:40:03.304009+0100 mgr.pve2-prg2a (mgr.91575377) 118749 :
cluster [DBG] pgmap v119002: 1737 pgs: 3 active+clean+laggy, 3
active+clean+scrubbing+deep, 1731 active+clean; 3.3 TiB data, 7.8 TiB used,
21 TiB / 29 TiB avail; 101 KiB/s rd, 1.9 MiB/s wr, 198 op/s

2020-12-10T21:40:05.316233+0100 mgr.pve2-prg2a (mgr.91575377) 118750 :
cluster [DBG] pgmap v119003: 1737 pgs: 6 active+clean+laggy, 3
active+clean+scrubbing+deep, 1728 active+clean; 3.3 TiB data, 7.8 TiB used,
21 TiB / 29 TiB avail; 150 KiB/s rd, 3.0 MiB/s wr, 249 op/s

2020-12-10T21:40:07.319643+0100 mgr.pve2-prg2a (mgr.91575377) 118751 :
cluster [DBG] pgmap v119004: 1737 pgs: 6 active+clean+laggy, 3
active+clean+scrubbing+deep, 1728 active+clean; 3.3 TiB data, 7.8 TiB used,
21 TiB / 29 TiB avail; 142 KiB/s rd, 2.3 MiB/s wr, 212 op/s





2020-12-10T21:40:15.523134+0100 mon.pve1-prg2a (mon.0) 125943 : cluster
[DBG] osd.4 reported failed by osd.24

2020-12-10T21:40:15.523325+0100 mon.pve1-prg2a (mon.0) 125944 : cluster
[DBG] osd.39 reported failed by osd.24

2020-12-10T21:40:16.112299+0100 mon.pve1-prg2a (mon.0) 125946 : cluster
[WRN] Health check failed: 0 slow ops, oldest one blocked for 32 sec, osd.8
has slow ops (SLOW_OPS)

2020-12-10T21:40:16.202867+0100 mon.pve1-prg2a (mon.0) 125947 : cluster
[DBG] osd.4 reported failed by osd.34

2020-12-10T21:40:16.202986+0100 mon.pve1-prg2a (mon.0) 125948 : cluster
[INF] osd.4 failed (root=default,host=pve1-prg2a) (2 reporters from
different host after 24.000267 >= grace 22.361677)

2020-12-10T21:40:16.373925+0100 mon.pve1-prg2a (mon.0) 125949 : cluster
[DBG] osd.39 reported failed by osd.6

2020-12-10T21:40:16.865608+0100 mon.pve1-prg2a (mon.0) 125951 : cluster
[DBG] osd.39 reported failed by osd.8

2020-12-10T21:40:17.125917+0100 mon.pve1-prg2a (mon.0) 125952 : cluster
[WRN] Health check failed: 1 osds down (OSD_DOWN)

2020-12-10T21:40:17.139006+0100 mon.pve1-prg2a (mon.0) 125953 : cluster
[DBG] osdmap e12819: 40 total, 39 up, 30 in

2020-12-10T21:40:17.140248+0100 mon.pve1-prg2a (mon.0) 125954 : cluster
[DBG] osd.39 reported failed by osd.21

2020-12-10T21:40:17.344244+0100 mgr.pve2-prg2a (mgr.91575377) 118757 :
cluster [DBG] pgmap v119012: 1737 pgs: 9 peering, 61 stale+active+clean, 1
active+clean+scrubbing+deep, 7 active+clean+laggy, 1659 active+clean; 3.3
TiB data, 7.8 TiB used, 14 TiB / 22 TiB avail; 44 KiB/s rd, 2.5 MiB/s wr,
107 op/s

2020-12-10T21:40:17.378069+0100 mon.pve1-prg2a (mon.0) 125955 : cluster
[DBG] osd.39 reported failed by osd.26

2020-12-10T21:40:17.424429+0100 mon.pve1-prg2a (mon.0) 125956 : cluster
[DBG] osd.39 reported failed by osd.18

2020-12-10T21:40:17.829447+0100 mon.pve1-prg2a (mon.0) 125957 : cluster
[DBG] osd.39 reported failed by osd.36

2020-12-10T21:40:17.847373+0100 mon.pve1-prg2a (mon.0) 125958 : cluster
[DBG] osd.39 reported failed by osd.1

2020-12-10T21:40:17.858371+0100 mon.pve1-prg2a (mon.0) 125959 : cluster
[DBG] osd.39 reported failed by osd.17

2020-12-10T21:40:17.915755+0100 mon.pve1-prg2a (mon.0) 125960 : c

[ceph-users] Re: Ceph benchmark tool (cbt)

[Mark Nelson]

This is what the "use_existing" flag is for (on by default).  It 
short-circuits initialize() which is what actually does the whole 
shutdown/creation/startup procedure.



https://github.com/ceph/cbt/blob/master/cluster/ceph.py#L149-L151


That is invoked before shutdown() and make_osds():


https://github.com/ceph/cbt/blob/master/cluster/ceph.py#L159

https://github.com/ceph/cbt/blob/master/cluster/ceph.py#L181


This is how a lot of folks are using cbt now with vstart or to test 
existing clusters deployed in the field.  When use_existing is disabled, 
cbt can build temporary multi-node bare metal test clusters for you by 
directly invoking the daemons via pdsh (it actually pre-dates 
ceph-deploy/ceph-ansible/etc).  The entire OSD structure is created in 
/tmp to hopefully convince people that this is only intended to be used 
for testing.  The code you are seeing to tear down old clusters and 
create new ones is for that path.  The advantage of this is that cbt has 
been very resilient to some of the churn over the years as different 
installer methods have come and gone.  The downside is that sometimes 
there are changes that require updates (for instance cbt created 
clusters still use msgr v1, which hopefully will change soon to support 
deploying crimson OSDs that require v2).



Mark


On 12/11/20 7:18 AM, Marc Roos wrote:

Just run the tool from a client that is not part of the ceph nodes. Than
it can do nothing, that you did not configure ceph to allow it to do ;)
Besides you should never run software from 'unknown' sources in an
environment where it can use 'admin' rights.
  



-Original Message-
To: ceph-users
Subject: [ceph-users] Ceph benchmark tool (cbt)

Hi all,

I want to benchmark my production cluster with cbt. I read a bit of the
code and I see something strange in it, for example, it's going to
create ceph-osd by it selves (
https://github.com/ceph/cbt/blob/master/cluster/ceph.py#L373) and also
shutdown the whole cluster!! (
https://github.com/ceph/cbt/blob/master/cluster/ceph.py#L212)

Is there any configuration to not do harmful things with the cluster and
for example just test the read_ahead_kb or simply stop some OSDs and
other things that can be reverted and not get the cluster fully down?!

Thanks.
___
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an
email to ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io


___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Scrubbing - osd down

[Miroslav Boháč]

Hi Igor,

thank you. Yes you are right.

It seems that the background removal is completed.

The correct way to fix it is "ceph-kvstore-tool bluestore-kv 
compact" to all OSD (one by one)?

Regards,
Miroslav

pá 11. 12. 2020 v 14:19 odesílatel Igor Fedotov  napsal:

> Hi Miroslav,
>
> haven't you performed massive data removal (PG migration) recently?
>
> If so you might want to apply manual DB compaction to your OSDs.
>
> The positive effect might be just temporary if background removals are
> still in progress though..
>
>
> See https://tracker.ceph.com/issues/47044 and the references for more
> info on the issue.
>
>
> Thanks,
>
> Igor
>
>
>
>
> On 12/11/2020 12:50 PM, Miroslav Boháč wrote:
> > Hi,
> >
> > I have a problem with crashing OSD daemons in our Ceph 15.2.6 cluster .
> The
> > problem was temporarily resolved by disabling scrub and deep-scrub. All
> PGs
> > are active+clean. After a few days I tried to enable scrubbing again, but
> > the problem persists. OSD with high latencies, PG laggy, osd not
> > responding, OSD was marked down and the cluster is not usable. Problem
> > appeared after failover when 10 OSD was marked as out.
> >
> > I will appreciate any help or advice, how to resolve this problem.
> >
> > Regards,
> > Miroslav
> >
> >
> >
> >
> >
> > 2020-12-10T21:39:57.721883+0100 osd.1 (osd.1) 5 : cluster [DBG] 18.11
> > deep-scrub starts
> >
> > 2020-12-10T21:39:57.880861+0100 osd.1 (osd.1) 6 : cluster [DBG] 18.11
> > deep-scrub ok
> >
> > 2020-12-10T21:39:58.713422+0100 osd.1 (osd.1) 7 : cluster [DBG] 43.5
> scrub
> > starts
> >
> > 2020-12-10T21:39:58.719372+0100 osd.1 (osd.1) 8 : cluster [DBG] 43.5
> scrub
> > ok
> >
> > 2020-12-10T21:39:59.296962+0100 mgr.pve2-prg2a (mgr.91575377) 118746 :
> > cluster [DBG] pgmap v119000: 1737 pgs: 3 active+clean+laggy, 3
> > active+clean+scrubbing+deep, 1731 active+clean; 3.3 TiB data, 7.8 TiB
> used,
> > 21 TiB / 29 TiB avail; 117 KiB/s rd, 2.5 MiB/s wr, 269 op/s
> >
> > 2020-12-10T21:40:00.088421+0100 osd.29 (osd.29) 74 : cluster [DBG] 1.13b
> > deep-scrub starts
> >
> > 2020-12-10T21:40:01.300373+0100 mgr.pve2-prg2a (mgr.91575377) 118747 :
> > cluster [DBG] pgmap v119001: 1737 pgs: 3 active+clean+laggy, 3
> > active+clean+scrubbing+deep, 1731 active+clean; 3.3 TiB data, 7.8 TiB
> used,
> > 21 TiB / 29 TiB avail; 101 KiB/s rd, 1.9 MiB/s wr, 202 op/s
> >
> > 2020-12-10T21:40:02.681058+0100 osd.34 (osd.34) 13 : cluster [DBG] 1.a2
> > deep-scrub ok
> >
> > 2020-12-10T21:40:03.304009+0100 mgr.pve2-prg2a (mgr.91575377) 118749 :
> > cluster [DBG] pgmap v119002: 1737 pgs: 3 active+clean+laggy, 3
> > active+clean+scrubbing+deep, 1731 active+clean; 3.3 TiB data, 7.8 TiB
> used,
> > 21 TiB / 29 TiB avail; 101 KiB/s rd, 1.9 MiB/s wr, 198 op/s
> >
> > 2020-12-10T21:40:05.316233+0100 mgr.pve2-prg2a (mgr.91575377) 118750 :
> > cluster [DBG] pgmap v119003: 1737 pgs: 6 active+clean+laggy, 3
> > active+clean+scrubbing+deep, 1728 active+clean; 3.3 TiB data, 7.8 TiB
> used,
> > 21 TiB / 29 TiB avail; 150 KiB/s rd, 3.0 MiB/s wr, 249 op/s
> >
> > 2020-12-10T21:40:07.319643+0100 mgr.pve2-prg2a (mgr.91575377) 118751 :
> > cluster [DBG] pgmap v119004: 1737 pgs: 6 active+clean+laggy, 3
> > active+clean+scrubbing+deep, 1728 active+clean; 3.3 TiB data, 7.8 TiB
> used,
> > 21 TiB / 29 TiB avail; 142 KiB/s rd, 2.3 MiB/s wr, 212 op/s
> >
> >
> >
> >
> >
> > 2020-12-10T21:40:15.523134+0100 mon.pve1-prg2a (mon.0) 125943 : cluster
> > [DBG] osd.4 reported failed by osd.24
> >
> > 2020-12-10T21:40:15.523325+0100 mon.pve1-prg2a (mon.0) 125944 : cluster
> > [DBG] osd.39 reported failed by osd.24
> >
> > 2020-12-10T21:40:16.112299+0100 mon.pve1-prg2a (mon.0) 125946 : cluster
> > [WRN] Health check failed: 0 slow ops, oldest one blocked for 32 sec,
> osd.8
> > has slow ops (SLOW_OPS)
> >
> > 2020-12-10T21:40:16.202867+0100 mon.pve1-prg2a (mon.0) 125947 : cluster
> > [DBG] osd.4 reported failed by osd.34
> >
> > 2020-12-10T21:40:16.202986+0100 mon.pve1-prg2a (mon.0) 125948 : cluster
> > [INF] osd.4 failed (root=default,host=pve1-prg2a) (2 reporters from
> > different host after 24.000267 >= grace 22.361677)
> >
> > 2020-12-10T21:40:16.373925+0100 mon.pve1-prg2a (mon.0) 125949 : cluster
> > [DBG] osd.39 reported failed by osd.6
> >
> > 2020-12-10T21:40:16.865608+0100 mon.pve1-prg2a (mon.0) 125951 : cluster
> > [DBG] osd.39 reported failed by osd.8
> >
> > 2020-12-10T21:40:17.125917+0100 mon.pve1-prg2a (mon.0) 125952 : cluster
> > [WRN] Health check failed: 1 osds down (OSD_DOWN)
> >
> > 2020-12-10T21:40:17.139006+0100 mon.pve1-prg2a (mon.0) 125953 : cluster
> > [DBG] osdmap e12819: 40 total, 39 up, 30 in
> >
> > 2020-12-10T21:40:17.140248+0100 mon.pve1-prg2a (mon.0) 125954 : cluster
> > [DBG] osd.39 reported failed by osd.21
> >
> > 2020-12-10T21:40:17.344244+0100 mgr.pve2-prg2a (mgr.91575377) 118757 :
> > cluster [DBG] pgmap v119012: 1737 pgs: 9 peering, 61 stale+active+clean,
> 1
> > active+clean+scrubbing+deep, 7 active+clean+laggy, 1659 active+clean; 3.3
> > TiB data, 7.8 TiB used

[ceph-users] Re: Scrubbing - osd down

[Igor Fedotov]

Miroslav,

On 12/11/2020 4:57 PM, Miroslav Boháč wrote:

Hi Igor,

thank you. Yes you are right.

It seems that the background removal is completed.
you can inspect "numpg_removing" performance counter to make sure it's 
been completed.


The correct way to fix it is "ceph-kvstore-tool bluestore-kv 
 compact" to all OSD (one by one)?


Right.

You can want to try online compaction: ceph tell osd.X compact

I'm still not sure the latter is effective in fixing the issue though. 
Will appreciate if you share your experience on that.



Regards,
Miroslav

pá 11. 12. 2020 v 14:19 odesílatel Igor Fedotov > napsal:


Hi Miroslav,

haven't you performed massive data removal (PG migration) recently?

If so you might want to apply manual DB compaction to your OSDs.

The positive effect might be just temporary if background removals
are
still in progress though..


See https://tracker.ceph.com/issues/47044
 and the references for more
info on the issue.


Thanks,

Igor




On 12/11/2020 12:50 PM, Miroslav Boháč wrote:
> Hi,
>
> I have a problem with crashing OSD daemons in our Ceph 15.2.6
cluster . The
> problem was temporarily resolved by disabling scrub and
deep-scrub. All PGs
> are active+clean. After a few days I tried to enable scrubbing
again, but
> the problem persists. OSD with high latencies, PG laggy, osd not
> responding, OSD was marked down and the cluster is not usable.
Problem
> appeared after failover when 10 OSD was marked as out.
>
> I will appreciate any help or advice, how to resolve this problem.
>
> Regards,
> Miroslav
>
>
>
>
>
> 2020-12-10T21:39:57.721883+0100 osd.1 (osd.1) 5 : cluster [DBG]
18.11
> deep-scrub starts
>
> 2020-12-10T21:39:57.880861+0100 osd.1 (osd.1) 6 : cluster [DBG]
18.11
> deep-scrub ok
>
> 2020-12-10T21:39:58.713422+0100 osd.1 (osd.1) 7 : cluster [DBG]
43.5 scrub
> starts
>
> 2020-12-10T21:39:58.719372+0100 osd.1 (osd.1) 8 : cluster [DBG]
43.5 scrub
> ok
>
> 2020-12-10T21:39:59.296962+0100 mgr.pve2-prg2a (mgr.91575377)
118746 :
> cluster [DBG] pgmap v119000: 1737 pgs: 3 active+clean+laggy, 3
> active+clean+scrubbing+deep, 1731 active+clean; 3.3 TiB data,
7.8 TiB used,
> 21 TiB / 29 TiB avail; 117 KiB/s rd, 2.5 MiB/s wr, 269 op/s
>
> 2020-12-10T21:40:00.088421+0100 osd.29 (osd.29) 74 : cluster
[DBG] 1.13b
> deep-scrub starts
>
> 2020-12-10T21:40:01.300373+0100 mgr.pve2-prg2a (mgr.91575377)
118747 :
> cluster [DBG] pgmap v119001: 1737 pgs: 3 active+clean+laggy, 3
> active+clean+scrubbing+deep, 1731 active+clean; 3.3 TiB data,
7.8 TiB used,
> 21 TiB / 29 TiB avail; 101 KiB/s rd, 1.9 MiB/s wr, 202 op/s
>
> 2020-12-10T21:40:02.681058+0100 osd.34 (osd.34) 13 : cluster
[DBG] 1.a2
> deep-scrub ok
>
> 2020-12-10T21:40:03.304009+0100 mgr.pve2-prg2a (mgr.91575377)
118749 :
> cluster [DBG] pgmap v119002: 1737 pgs: 3 active+clean+laggy, 3
> active+clean+scrubbing+deep, 1731 active+clean; 3.3 TiB data,
7.8 TiB used,
> 21 TiB / 29 TiB avail; 101 KiB/s rd, 1.9 MiB/s wr, 198 op/s
>
> 2020-12-10T21:40:05.316233+0100 mgr.pve2-prg2a (mgr.91575377)
118750 :
> cluster [DBG] pgmap v119003: 1737 pgs: 6 active+clean+laggy, 3
> active+clean+scrubbing+deep, 1728 active+clean; 3.3 TiB data,
7.8 TiB used,
> 21 TiB / 29 TiB avail; 150 KiB/s rd, 3.0 MiB/s wr, 249 op/s
>
> 2020-12-10T21:40:07.319643+0100 mgr.pve2-prg2a (mgr.91575377)
118751 :
> cluster [DBG] pgmap v119004: 1737 pgs: 6 active+clean+laggy, 3
> active+clean+scrubbing+deep, 1728 active+clean; 3.3 TiB data,
7.8 TiB used,
> 21 TiB / 29 TiB avail; 142 KiB/s rd, 2.3 MiB/s wr, 212 op/s
>
>
>
>
>
> 2020-12-10T21:40:15.523134+0100 mon.pve1-prg2a (mon.0) 125943 :
cluster
> [DBG] osd.4 reported failed by osd.24
>
> 2020-12-10T21:40:15.523325+0100 mon.pve1-prg2a (mon.0) 125944 :
cluster
> [DBG] osd.39 reported failed by osd.24
>
> 2020-12-10T21:40:16.112299+0100 mon.pve1-prg2a (mon.0) 125946 :
cluster
> [WRN] Health check failed: 0 slow ops, oldest one blocked for 32
sec, osd.8
> has slow ops (SLOW_OPS)
>
> 2020-12-10T21:40:16.202867+0100 mon.pve1-prg2a (mon.0) 125947 :
cluster
> [DBG] osd.4 reported failed by osd.34
>
> 2020-12-10T21:40:16.202986+0100 mon.pve1-prg2a (mon.0) 125948 :
cluster
> [INF] osd.4 failed (root=default,host=pve1-prg2a) (2 reporters from
> different host after 24.000267 >= grace 22.361677)
>
> 2020-12-10T21:40:16.373925+0100 mon.pve1-prg2a (mon.0) 125949 :
cluster
> [DBG] osd.39 reported failed by osd.6
>
> 2020-12-10T21:40:16.865608+0100 mon.pve1-prg2a (mon.0) 125951 :
cl

[ceph-users] Re: mgr's stop responding, dropping out of cluster with _check_auth_rotating

[Wido den Hollander]

On 11/12/2020 00:12, David Orman wrote:

Hi Janek,

We realize this, we referenced that issue in our initial email. We do want
the metrics exposed by Ceph internally, and would prefer to work towards a
fix upstream. We appreciate the suggestion for a workaround, however!

Again, we're happy to provide whatever information we can that would be of
assistance. If there's some debug setting that is preferred, we are happy
to implement it, as this is currently a test cluster for us to work through
issues such as this one.



Have you tried disabling Prometheus just to see if this also fixes the 
issue for you?


Wido


David

On Thu, Dec 10, 2020 at 12:02 PM Janek Bevendorff <
janek.bevendo...@uni-weimar.de> wrote:


Do you have the prometheus module enabled? Turn that off, it's causing
issues. I replaced it with another ceph exporter from Github and almost
forgot about it.

Here's the relevant issue report:
https://tracker.ceph.com/issues/39264#change-179946

On 10/12/2020 16:43, Welby McRoberts wrote:

Hi Folks

We've noticed that in a cluster of 21 nodes (5 mgrs&mons & 504 OSDs with

24

per node) that the mgr's are, after a non specific period of time,

dropping

out of the cluster. The logs only show the following:

debug 2020-12-10T02:02:50.409+ 7f1005840700  0 log_channel(cluster)

log

[DBG] : pgmap v14163: 4129 pgs: 4129 active+clean; 10 GiB data, 31 TiB
used, 6.3 PiB / 6.3 PiB avail
debug 2020-12-10T03:20:59.223+ 7f10624eb700 -1 monclient:
_check_auth_rotating possible clock skew, rotating keys expired way too
early (before 2020-12-10T02:20:59.226159+)
debug 2020-12-10T03:21:00.223+ 7f10624eb700 -1 monclient:
_check_auth_rotating possible clock skew, rotating keys expired way too
early (before 2020-12-10T02:21:00.226310+)

The _check_auth_rotating repeats approximately every second. The

instances

are all syncing their time with NTP and have no issues on that front. A
restart of the mgr fixes the issue.

It appears that this may be related to

https://tracker.ceph.com/issues/39264.

The suggestion seems to be to disable prometheus metrics, however, this
obviously isn't realistic for a production environment where metrics are
critical for operations.

Please let us know what additional information we can provide to assist

in

resolving this critical issue.

Cheers
Welby
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io


___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io


___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: mgr's stop responding, dropping out of cluster with _check_auth_rotating

[David Orman]

No, as the number of responses we've seen in the mailing lists and on the
bug report(s) have indicated it fixed the situation, we didn't proceed down
that path (it seemed highly probable it would resolve things). If it's of
additional value, we can disable the module temporarily to see if the
problem no longer presents itself, but our intent would not be to continue
to leave the module disabled and instead work towards resolution of the
issue at hand.

Let us know if disabling this module would assist in troubleshooting, and
we're happy to do so.

FWIW - we've also built a container with all of the debuginfo packages and
gdb setup to inspect the unresponsive ceph-mgr process, but our
understanding of ceph's internal workings is not deep enough to determine
why it appears to be deadlocking. That said, we welcome any requests for
any additional information we can provide to assist in determining the
cause/implementation of a solution.

David

On Fri, Dec 11, 2020 at 8:10 AM Wido den Hollander  wrote:

>
>
> On 11/12/2020 00:12, David Orman wrote:
> > Hi Janek,
> >
> > We realize this, we referenced that issue in our initial email. We do
> want
> > the metrics exposed by Ceph internally, and would prefer to work towards
> a
> > fix upstream. We appreciate the suggestion for a workaround, however!
> >
> > Again, we're happy to provide whatever information we can that would be
> of
> > assistance. If there's some debug setting that is preferred, we are happy
> > to implement it, as this is currently a test cluster for us to work
> through
> > issues such as this one.
> >
>
> Have you tried disabling Prometheus just to see if this also fixes the
> issue for you?
>
> Wido
>
> > David
> >
> > On Thu, Dec 10, 2020 at 12:02 PM Janek Bevendorff <
> > janek.bevendo...@uni-weimar.de> wrote:
> >
> >> Do you have the prometheus module enabled? Turn that off, it's causing
> >> issues. I replaced it with another ceph exporter from Github and almost
> >> forgot about it.
> >>
> >> Here's the relevant issue report:
> >> https://tracker.ceph.com/issues/39264#change-179946
> >>
> >> On 10/12/2020 16:43, Welby McRoberts wrote:
> >>> Hi Folks
> >>>
> >>> We've noticed that in a cluster of 21 nodes (5 mgrs&mons & 504 OSDs
> with
> >> 24
> >>> per node) that the mgr's are, after a non specific period of time,
> >> dropping
> >>> out of the cluster. The logs only show the following:
> >>>
> >>> debug 2020-12-10T02:02:50.409+ 7f1005840700  0 log_channel(cluster)
> >> log
> >>> [DBG] : pgmap v14163: 4129 pgs: 4129 active+clean; 10 GiB data, 31 TiB
> >>> used, 6.3 PiB / 6.3 PiB avail
> >>> debug 2020-12-10T03:20:59.223+ 7f10624eb700 -1 monclient:
> >>> _check_auth_rotating possible clock skew, rotating keys expired way too
> >>> early (before 2020-12-10T02:20:59.226159+)
> >>> debug 2020-12-10T03:21:00.223+ 7f10624eb700 -1 monclient:
> >>> _check_auth_rotating possible clock skew, rotating keys expired way too
> >>> early (before 2020-12-10T02:21:00.226310+)
> >>>
> >>> The _check_auth_rotating repeats approximately every second. The
> >> instances
> >>> are all syncing their time with NTP and have no issues on that front. A
> >>> restart of the mgr fixes the issue.
> >>>
> >>> It appears that this may be related to
> >> https://tracker.ceph.com/issues/39264.
> >>> The suggestion seems to be to disable prometheus metrics, however, this
> >>> obviously isn't realistic for a production environment where metrics
> are
> >>> critical for operations.
> >>>
> >>> Please let us know what additional information we can provide to assist
> >> in
> >>> resolving this critical issue.
> >>>
> >>> Cheers
> >>> Welby
> >>> ___
> >>> ceph-users mailing list -- ceph-users@ceph.io
> >>> To unsubscribe send an email to ceph-users-le...@ceph.io
> >> ___
> >> ceph-users mailing list -- ceph-users@ceph.io
> >> To unsubscribe send an email to ceph-users-le...@ceph.io
> >>
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
> >
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Slow Replication on Campus

[Vikas Rana]

Hi Friends,

 

We have 2 Ceph clusters on campus and we setup the second cluster as the DR
solution.

The images on the DR side are always behind the master.

 

Ceph Version : 12.2.11 

 

 

VMWARE_LUN0:

  global_id:   23460954-6986-4961-9579-0f2a1e58e2b2

  state:   up+replaying

  description: replaying, master_position=[object_number=2632711,
tag_tid=24, entry_tid=1967382595], mirror_position=[object_number=1452837,
tag_tid=24, entry_tid=456440697], entries_behind_master=1510941898

  last_update: 2020-11-30 14:13:38

 

VMWARE_LUN1:

  global_id:   cb579579-13b0-4522-b65f-c64ec44cbfaf

  state:   up+replaying

  description: replaying, master_position=[object_number=1883943,
tag_tid=28, entry_tid=1028822927], mirror_position=[object_number=1359161,
tag_tid=28, entry_tid=358296085], entries_behind_master=670526842

  last_update: 2020-11-30 14:13:33

 

Any suggestion on tuning or any parameters we can set on RBD-mirror to speed
up the replication. Both cluster have very little activity.

 

 

Appreciate your help.

 

Thanks,

-Vikas

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Incomplete PG due to primary OSD crashing during EC backfill - get_hash_info: Mismatch of total_chunk_size 0

[Byrne, Thomas (STFC,RAL,SC)]

After confirming that the corruption was limited to a single object, we deleted 
the object (first via radosgw-admin, and the via a rados rm), and restarted the 
new OSD in the set. The backfill has continued past the point of the original 
crash, so things are looking promising.

I'm still concerned about how we ended up in this state, clearly there was a 
very sad disk involved, but it's concerning that the object was corrupted 
across all OSDs in the set, and also concerning that the user had no idea the 
file was corrupted (200 HTTP codes returned).

I'm going dig further into the logs of the original primary to see if there are 
anything that looks suspicious around the time of the original write, but for 
now I'm glad to have a happy cluster in time for the weekend!

Cheers,
Tom

> -Original Message-
> From: Byrne, Thomas (STFC,RAL,SC) 
> Sent: 10 December 2020 23:34
> To: 'ceph-users' 
> Cc: Dan van der Ster 
> Subject: [ceph-users] Re: Incomplete PG due to primary OSD crashing during
> EC backfill - get_hash_info: Mismatch of total_chunk_size 0
> 
> A few more things of note after more poking with the help of Dan vdS.
> 
> 1) The object that the backfill is crashing on has an mtime of a few minutes
> before the original primary died this morning, and a 'rados get' gives an
> input/output error. So it looks like a new object that was possibly corrupted 
> by
> the dying primary OSD. I can't see any disk I/O error in any of the PGs OSD
> logs when trying the 'get', but I do see this error in most of the OSDs logs:
> 
> 2020-12-10 23:22:31.840 7fc7161e3700  0 osd.4134 pg_epoch: 1162547
> pg[11.214s8( v 1162547'714924 (1162114'711864,1162547'714924] local-
> lis/les=1162304/1162305 n=133402 ec=1069520/992 lis/c 1162304/1125301
> les/c/f 1162305/1125302/257760 1162303/1162304/1162301)
> [2147483647,1708,2099,1346,4309,777,5098,4501,4134,217,4643]p1708(1)
> r=8 lpr=1162304 pi=[1125301,1162304)/2 luod=0'0 crt=1162547'714924
> active mbc={}] get_hash_info: Mismatch of total_chunk_size 0
> 2020-12-10 23:22:31.840 7fc7161e3700 -1 log_channel(cluster) log [ERR] :
> Corruption detected: object 11:28447b4a:::962de230-ed6c-44f2-ab02-
> 788c52ea6a82.3210530112.122__multipart_201%2fin5%2fexp_4-05-
> 737%2fprocessed%2fspe%2fsqw_187570.nxspe.2~bgZPo_rC64ZXJWKyTfdn4dI
> ApqLNDPp.22:head is missing hash_info
> 
> This error was present in OSD logs: 1708, 2099, 1346, 4309, 777, 5098, 4501,
> 4134, and absent in: 217, 4643 (possibly because they are unused parity
> shards?). Checking on one of the FileStore OSDs that returned the error
> message, the underlying file is present and the correct size at least.
> 
> I'm checking all objects in the PG now for corruption, I'm only 25% through
> the 133385 objects in the PG, but that object is the only corrupted one I've
> seen so far, so hopefully it is an isolated corruption. If so, I can possibly 
> try
> deleting the problematic object and seeing if the backfill can continue.
> 
> 2)  This PG is a mix of FileStore and BlueStore OSDs, all 14.2.9. The original
> primary that died (1466) was FileStore. Bluestore: 1708, 4309, 5098, 4501,
> 4134, 4643, Filestore: 2099, 1346, 777, 217.
> 
> PG query for reference: https://pastebin.com/ZUUH2mQ6
> 
> Cheers,
> Tom
> 
> > -Original Message-
> > From: Byrne, Thomas (STFC,RAL,SC) 
> > Sent: 10 December 2020 18:40
> > To: 'ceph-users' 
> > Subject: [ceph-users] Incomplete PG due to primary OSD crashing during
> > EC backfill - get_hash_info: Mismatch of total_chunk_size 0
> >
> > Hi all,
> >
> > Got an odd issue that I'm not sure how to solve on our Nautilus 14.2.9
> > EC cluster.
> >
> > The primary OSD of an EC 8+3 PG died this morning with a very sad disk
> > (thousands of pending sectors). After the down out interval a new 'up'
> > primary was assigned and the backfill started. Twenty minutes later
> > the acting primary (not the new 'up' primary) started crashing with a
> "get_hash_info:
> > Mismatch of total_chunk_size 0" error (see log below)
> >
> > This crash always happens at the same object, with different acting
> > primaries, and with a different new 'up' primary. I can't see anything
> > in the logs that points to a particular OSD being the issue, so I
> > suspect there is a corrupted object in the PG that is causing issues,
> > but I'm not sure how to dig into this further. The PG is currently
> > active (but degraded), but only whilst nobackfill or noout are set
> > (+turning the new OSD off), and if the flags are unset the backfill
> > will eventually crash enough OSDs to render the PG incomplete, which
> > is not ideal. I would appreciate being able to resolve this so I can
> > go back to letting Ceph deal with down OSDs itself :)
> >
> > Does anyone have some pointers on how to dig into or resolve this?
> > Happy to create a tracker ticket and post more logs if this looks like a 
> > bug.
> >
> > Thanks,
> > Tom
> >
> > OSD log with debug_osd=20 (preamble cut from subsequent lines in an
> > attempt t

[ceph-users] Debian repo for ceph-iscsi

[Chris Palmer]

I just went to setup an iscsi gateway on a Debian Buster / Octopus 
cluster and hit a brick wall with packages. I had perhaps naively 
assumed they were in with the rest. Now I understand that it can exist 
separately, but then so can RGW.


I found some ceph-iscsi rpm builds for Centos, but nothing for Debian. 
Are they around somewhere? The prerequisite packages 
python-rtslib-2.1.fb68 and tcmu-runner-1.4.0 also don't seem to be 
readily available for Debian.


Has anyone done this for Debian?

Thanks, Chris
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: CephFS max_file_size

[Patrick Donnelly]

Hi Mark,

On Fri, Dec 11, 2020 at 4:21 AM Mark Schouten  wrote:
> There is a default limit of 1TiB for the max_file_size in CephFS. I altered 
> that to 2TiB, but I now got a request for storing a file up to 7TiB.
>
> I'd expect the limit to be there for a reason, but what is the risk of 
> setting that value to say 10TiB?

There is no known downside. Let us know how it goes!

--
Patrick Donnelly, Ph.D.
He / Him / His
Principal Software Engineer
Red Hat Sunnyvale, CA
GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: CephFS max_file_size

[Adam Tygart]

I've had this set to 16TiB for several years now.

I've not seen any ill effects.

--
Adam

On Fri, Dec 11, 2020 at 12:56 PM Patrick Donnelly  wrote:
>
> Hi Mark,
>
> On Fri, Dec 11, 2020 at 4:21 AM Mark Schouten  wrote:
> > There is a default limit of 1TiB for the max_file_size in CephFS. I altered 
> > that to 2TiB, but I now got a request for storing a file up to 7TiB.
> >
> > I'd expect the limit to be there for a reason, but what is the risk of 
> > setting that value to say 10TiB?
>
> There is no known downside. Let us know how it goes!
>
> --
> Patrick Donnelly, Ph.D.
> He / Him / His
> Principal Software Engineer
> Red Hat Sunnyvale, CA
> GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: CephFS max_file_size

[Paul Mezzanini]

>From how I understand it, that setting is a rev-limiter to prevent users from 
>creating HUGE sparse files and then wasting cluster resources firing off 
>deletes.

We have ours set to 32T and haven't seen any issues with large files.

--
Paul Mezzanini
Sr Systems Administrator / Engineer, Research Computing
Information & Technology Services
Finance & Administration
Rochester Institute of Technology
o:(585) 475-3245 | pfm...@rit.edu

CONFIDENTIALITY NOTE: The information transmitted, including attachments, is
intended only for the person(s) or entity to which it is addressed and may
contain confidential and/or privileged material. Any review, retransmission,
dissemination or other use of, or taking of any action in reliance upon this
information by persons or entities other than the intended recipient is
prohibited. If you received this in error, please contact the sender and
destroy any copies of this information.



From: Adam Tygart 
Sent: Friday, December 11, 2020 1:59 PM
To: Mark Schouten
Cc: ceph-users; Patrick Donnelly
Subject: [ceph-users] Re: CephFS max_file_size

I've had this set to 16TiB for several years now.

I've not seen any ill effects.

--
Adam

On Fri, Dec 11, 2020 at 12:56 PM Patrick Donnelly  wrote:
>
> Hi Mark,
>
> On Fri, Dec 11, 2020 at 4:21 AM Mark Schouten  wrote:
> > There is a default limit of 1TiB for the max_file_size in CephFS. I altered 
> > that to 2TiB, but I now got a request for storing a file up to 7TiB.
> >
> > I'd expect the limit to be there for a reason, but what is the risk of 
> > setting that value to say 10TiB?
>
> There is no known downside. Let us know how it goes!
>
> --
> Patrick Donnelly, Ph.D.
> He / Him / His
> Principal Software Engineer
> Red Hat Sunnyvale, CA
> GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] diskprediction_local to be retired or fixed or??

[Harry G. Coin]

Any idea whether 'diskprediction_local' will ever work in containers? 
I'm running 15.2.7 which contains a dependency on scikit-learn v 0.19.2
which isn't in the container.  It's been throwing that error for a year
now on all the octopus container versions I tried.  It used to be on the
baremetal versions

pip3 install --upgrade scikit-learn==0.19.2

would fix it, but that seems to break the mgr container.

Thanks

Harry


___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Debian repo for ceph-iscsi

[Reed Dier]

I know this isn't what you asked for, but I do know that Canonical is building 
this package for focal and up.
While not Buster, could possibly be a compromise to move things forward without 
huge plumbing changes between Debian and Ubuntu.

You may also be able to hack and slash your way through to Debian pulling that 
.deb and trying to make it work.

Thats the only way I've seen it available in Ubuntu thus far.

Reed

> On Dec 11, 2020, at 12:11 PM, Chris Palmer  wrote:
> 
> I just went to setup an iscsi gateway on a Debian Buster / Octopus cluster 
> and hit a brick wall with packages. I had perhaps naively assumed they were 
> in with the rest. Now I understand that it can exist separately, but then so 
> can RGW.
> 
> I found some ceph-iscsi rpm builds for Centos, but nothing for Debian. Are 
> they around somewhere? The prerequisite packages python-rtslib-2.1.fb68 and 
> tcmu-runner-1.4.0 also don't seem to be readily available for Debian.
> 
> Has anyone done this for Debian?
> 
> Thanks, Chris
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io



smime.p7s
Description: S/MIME cryptographic signature
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] MON: global_init: error reading config file.

[Oscar Segarra]

Hi,

In my environment I have a single node and I'm trying to run a ceph monitor
as a docker container using a kv store.

Version: Octopus (stable-5.0)

2020-12-12 00:24:28  /opt/ceph-container/bin/entrypoint.sh: STAYALIVE:
container will not die if a command fails.,
2020-12-12 00:24:28  /opt/ceph-container/bin/entrypoint.sh: 'auth' key
already exists,
2020-12-12 00:24:28  /opt/ceph-container/bin/entrypoint.sh: 'global' key
already exists,
2020-12-12 00:24:28  /opt/ceph-container/bin/entrypoint.sh: 'mon' key
already exists,
2020-12-12 00:24:28  /opt/ceph-container/bin/entrypoint.sh: 'mds' key
already exists,
2020-12-12 00:24:28  /opt/ceph-container/bin/entrypoint.sh: 'osd' key
already exists,
2020-12-12 00:24:28  /opt/ceph-container/bin/entrypoint.sh: 'client' key
already exists,
2020-12-12 00:24:28  /opt/ceph-container/bin/entrypoint.sh: Adding Mon Host
- vdicnode01.,
0.0.0.0,
vdicnode01,
true,
2020-12-12 00:24:28  /opt/ceph-container/bin/entrypoint.sh: Configuration
found for cluster ceph. Writing to disk.,
true,
2020-12-12T00:24:29+01:00 vdicnode01.local /tmp/gztmpGeLTQkuFB/confd[139]:
INFO Backend set to etcd,
2020-12-12T00:24:29+01:00 vdicnode01.local /tmp/gztmpGeLTQkuFB/confd[139]:
INFO Starting confd,
2020-12-12T00:24:29+01:00 vdicnode01.local /tmp/gztmpGeLTQkuFB/confd[139]:
INFO Backend source(s) set to http://127.0.0.1:2379,
2020-12-12 00:24:29  /opt/ceph-container/bin/entrypoint.sh: Adding
bootstrap keyrings.,
2020-12-12 00:24:30  /opt/ceph-container/bin/entrypoint.sh: Adding
mon/admin Keyrings.,
2020-12-12 00:24:30  /opt/ceph-container/bin/entrypoint.sh: Trying to get
the most recent monmap...,
Error initializing cluster client: InvalidArgumentError('RADOS invalid
argument (error calling conf_read_file)',),
2020-12-12 00:24:30  /opt/ceph-container/bin/entrypoint.sh: Peers not
found, using initial monmap.,
2020-12-12 00:24:30  /opt/ceph-container/bin/entrypoint.sh: Removing lock
for vdicnode01.,
PrevNode.Value: vdicnode01,
importing contents of /var/lib/ceph/bootstrap-osd/ceph.keyring into
/etc/ceph/ceph.mon.keyring,
importing contents of /var/lib/ceph/bootstrap-mds/ceph.keyring into
/etc/ceph/ceph.mon.keyring,
importing contents of /var/lib/ceph/bootstrap-rgw/ceph.keyring into
/etc/ceph/ceph.mon.keyring,
importing contents of /var/lib/ceph/bootstrap-rbd-mirror/ceph.keyring into
/etc/ceph/ceph.mon.keyring,
importing contents of /etc/ceph/ceph.client.admin.keyring into
/etc/ceph/ceph.mon.keyring,
global_init: error reading config file.,

And container dies (even the stayalive flag).

docker service create -d --network=host \
--mount type=bind,source=/var/lib/ceph,destination=/var/lib/ceph \
--mount type=bind,source=/var/log/ceph,destination=/var/log/ceph \
--mount type=bind,source=/etc/localtime,destination=/etc/localtime,readonly
\
-e KV_TYPE=etcd \
-e KV_IP=127.0.0.1 \
-e KV_PORT=2379 \
-e MON_IP=0.0.0.0 \
-e CEPH_PUBLIC_NETWORK=192.168.100.0/24 \
-e CEPH_CLUSTER_NETWORK=192.168.100.0/24 \
-e DEBUG=stayalive \
--name=vdicube_ceph_mon \
--mode=global \
--constraint node.labels.mon==1 \
ceph/daemon:v5.0.6-stable-5.0-octopus-centos-8 mon

Anyone has experimented the same issue?

Thanks a lot.
Ó
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io