[ceph-users] Nautilus RadosGW "One Zone" like AWS

[Florian Engelmann]

Hello,

is it already possible or planned to enable RadosGW to have a storage 
class like AWS does with "one zone" where the objects only exists in one 
zone and don't get mirrored to any other zone? eg. by creating the 
placement pool only in one zone?


If this feature isn't planned is it worth creating a feature request? 
What do you think about that feature?


All the best,
Florian


smime.p7s
Description: S/MIME cryptographic signature
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: list CephFS snapshots

[Lars Täuber]

Hi Frank,

thanks for you hint. The find for the inode is really fast. At least fast 
enough for me:
$ time find /mnt/point -inum 1093514215110 -print -quit
real0m3,009s
user0m0,037s
sys 0m0,032s

Cheers,
Lars


Tue, 17 Dec 2019 15:08:06 +
Frank Schilder  ==> Marc Roos , taeuber 
 :
> I think you can do a find for the inode (-inum n). At last I hope you can.
> 
> However, I vaguely remember that there was a thread where someone gave a 
> really nice MDS command for finding the path to an inode in no time.
> 
> Best regards,
> 
> =
> Frank Schilder
> AIT Risø Campus
> Bygning 109, rum S14
> 
> 
> From: Marc Roos 
> Sent: 17 December 2019 14:19:54
> To: Frank Schilder; taeuber
> Cc: ceph-users
> Subject: RE: [ceph-users] Re: list CephFS snapshots
> 
> Thanks, Good tip! If I do not know where I created these, is there a way
> to get their location in the filesystem? Or maybe a command that deletes
> by snapid?
> 
> 
> {
> "snapid": 54,
> "ino": 1099519875627,
> "stamp": "2017-09-13 21:21:35.769863",
> "name": "snap-20170913"
> },
> {
> "snapid": 153485,
> "ino": 1099519910289,
> "stamp": "2019-10-06 03:18:03.933510",
> "name": "snap-6"
> },
> {
> "snapid": 153489,
> "ino": 1099519910289,
> "stamp": "2019-10-07 03:21:03.218324",
> "name": "snap-7"
> },
> 
> 
> 
> -Original Message-
> Cc: ceph-users@ceph.io
> Subject: [ceph-users] Re: list CephFS snapshots
> 
> Have you tried "ceph daemon mds.NAME dump snaps" (available since
> mimic)?
> 
> =
> Frank Schilder
> AIT Risø Campus
> Bygning 109, rum S14
> 
> 
> From: Lars Täuber 
> Sent: 17 December 2019 12:32:34
> To: Stephan Mueller
> Cc: ceph-users@ceph.io
> Subject: [ceph-users] Re: list CephFS snapshots
> 
> Hi Michael,
> 
> thanks for your gist.
> This is at least a way to do it. But there are many directories in our
> cluster.
> The "find $1 -type d" lasts for about 90 minutes to find all 2.6 million
> directories.
> 
> Is there another (faster) way e.g. via mds?
> 
> Cheers,
> Lars
> 
> 
> Mon, 16 Dec 2019 17:03:41 +
> Stephan Mueller  ==> "taeu...@bbaw.de"
> , "ceph-users@ceph.io"  :
> > Hi Lars,
> >  
> > > Is there a mean to list all snapshots existing in a (subdir of)
> > > Cephfs?
> > > I can't use the find dommand to look for the ".snap" dirs.  
> >
> > You can, but you can't search for the '.snap' directories, you have to  
> 
> > append them to the directory like `find $cephFsDir/.snap` but I it's
> > better to use `ls` instead, to list all snapshots.
> >  
> > >
> > > I'd like to remove certain (or all) snapshots within a CephFS. But
> > > how do I find them?
> > >  
> >
> > I just created a gist for you that can do that:
> > https://gist.github.com/Devp00l/2473f5953d578f440fc71b3d602a9c23
> >
> > As you can see in the script, snapshots starting with an underscore
> > are filtered out as these directories belong to snapshots that were
> > created in upper directories and these underscore snapshots can't be
> > used for deletion.
> >
> > The deletion of a snapshot can be done by calling `rmdir`.
> >
> > But if you really want to manage CephFS snapshots easily take a look
> > at the dashboard, as we have integrated the snapshot and quota
> > management by now :)
> >
> > You can delete multiple snapshots of a directory or just create new
> > snapshots on a directory basis easily through the UI.
> >
> >
> > Stephan  
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: list CephFS snapshots

[Frank Schilder]

I found it, should have taken a note:

Command: rados -p  getxattr . 
parent | ceph-dencoder type inode_backtrace_t import - decode dump_json

Note:  is hex encoded, use 'printf "%x\n" INUM' to convert 
from the decimal numbers obtained with dump snaps.

Explanation: 


I don't have snapshots enabled and cannot test. Could you confirm to the 
mailing list that the above procedure will provide the path information all the 
way down to the actual snap dir, that is, contains entries ".snap" and 
"" at the top?

Best regards,

=
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14


From: Lars Täuber 
Sent: 18 December 2019 09:24:02
To: Frank Schilder
Cc: Marc Roos; ceph-users
Subject: Re: [ceph-users] Re: list CephFS snapshots

Hi Frank,

thanks for you hint. The find for the inode is really fast. At least fast 
enough for me:
$ time find /mnt/point -inum 1093514215110 -print -quit
real0m3,009s
user0m0,037s
sys 0m0,032s

Cheers,
Lars


Tue, 17 Dec 2019 15:08:06 +
Frank Schilder  ==> Marc Roos , taeuber 
 :
> I think you can do a find for the inode (-inum n). At last I hope you can.
>
> However, I vaguely remember that there was a thread where someone gave a 
> really nice MDS command for finding the path to an inode in no time.
>
> Best regards,
>
> =
> Frank Schilder
> AIT Risø Campus
> Bygning 109, rum S14
>
> 
> From: Marc Roos 
> Sent: 17 December 2019 14:19:54
> To: Frank Schilder; taeuber
> Cc: ceph-users
> Subject: RE: [ceph-users] Re: list CephFS snapshots
>
> Thanks, Good tip! If I do not know where I created these, is there a way
> to get their location in the filesystem? Or maybe a command that deletes
> by snapid?
>
>
> {
> "snapid": 54,
> "ino": 1099519875627,
> "stamp": "2017-09-13 21:21:35.769863",
> "name": "snap-20170913"
> },
> {
> "snapid": 153485,
> "ino": 1099519910289,
> "stamp": "2019-10-06 03:18:03.933510",
> "name": "snap-6"
> },
> {
> "snapid": 153489,
> "ino": 1099519910289,
> "stamp": "2019-10-07 03:21:03.218324",
> "name": "snap-7"
> },
>
>
>
> -Original Message-
> Cc: ceph-users@ceph.io
> Subject: [ceph-users] Re: list CephFS snapshots
>
> Have you tried "ceph daemon mds.NAME dump snaps" (available since
> mimic)?
>
> =
> Frank Schilder
> AIT Risø Campus
> Bygning 109, rum S14
>
> 
> From: Lars Täuber 
> Sent: 17 December 2019 12:32:34
> To: Stephan Mueller
> Cc: ceph-users@ceph.io
> Subject: [ceph-users] Re: list CephFS snapshots
>
> Hi Michael,
>
> thanks for your gist.
> This is at least a way to do it. But there are many directories in our
> cluster.
> The "find $1 -type d" lasts for about 90 minutes to find all 2.6 million
> directories.
>
> Is there another (faster) way e.g. via mds?
>
> Cheers,
> Lars
>
>
> Mon, 16 Dec 2019 17:03:41 +
> Stephan Mueller  ==> "taeu...@bbaw.de"
> , "ceph-users@ceph.io"  :
> > Hi Lars,
> >
> > > Is there a mean to list all snapshots existing in a (subdir of)
> > > Cephfs?
> > > I can't use the find dommand to look for the ".snap" dirs.
> >
> > You can, but you can't search for the '.snap' directories, you have to
>
> > append them to the directory like `find $cephFsDir/.snap` but I it's
> > better to use `ls` instead, to list all snapshots.
> >
> > >
> > > I'd like to remove certain (or all) snapshots within a CephFS. But
> > > how do I find them?
> > >
> >
> > I just created a gist for you that can do that:
> > https://gist.github.com/Devp00l/2473f5953d578f440fc71b3d602a9c23
> >
> > As you can see in the script, snapshots starting with an underscore
> > are filtered out as these directories belong to snapshots that were
> > created in upper directories and these underscore snapshots can't be
> > used for deletion.
> >
> > The deletion of a snapshot can be done by calling `rmdir`.
> >
> > But if you really want to manage CephFS snapshots easily take a look
> > at the dashboard, as we have integrated the snapshot and quota
> > management by now :)
> >
> > You can delete multiple snapshots of a directory or just create new
> > snapshots on a directory basis easily through the UI.
> >
> >
> > Stephan
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] radosgw - Etags suffixed with #x0e

[Ingo Reimann]

Hi,

We had a strange problem with some buckets. After a s3cmd sync, some objects 
got ETAGs with the suffix "#x0e". This rendered the XML output of "GET /" e.g. 
(s3cmd du) invalid. Unfortunately, this behaviour was not reproducable but 
could be fixed by "GET /{object}" + "PUT /{object}" (s3cmd get + s3cmd put).

I am not sure, how this appeared and how to avoid that. Just now, we have 
nautilus mons and osds with jewel radosgws. At the time of first appearence, 
also a nautilus gateway had been online, but the requests had been handled by 
both types.

Any ideas?

best regards,
Ingo

-- 
Ingo Reimann 
Teamleiter Technik 
[ https://www.dunkel.de/ ] 
Dunkel GmbH 
Philipp-Reis-Straße 2 
65795 Hattersheim 
Fon: +49 6190 889-100 
Fax: +49 6190 889-399 
eMail: supp...@dunkel.de 
http://www.Dunkel.de/   Amtsgericht Frankfurt/Main 
HRB: 37971 
Geschäftsführer: Axel Dunkel 
Ust-ID: DE 811622001
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: list CephFS snapshots

[Lars Täuber]

Hi Frank,

the command takes the metadata pool not the data pool as argument!

rados -p  getxattr $(printf "%x\n" ). parent | ceph-dencoder type inode_backtrace_t import - decode 
dump_json

The result is a not very human readable json output:
{
"ino": 1099511755110,
"ancestors": [
{
"dirino": 1099511946098,
"dname": "CCC",
"version": 1850022
},
{
"dirino": 1099511627776,
"dname": "BBB",
"version": 1847528
},
{
"dirino": 1,
"dname": "AAA",
"version": 222830
}
],
"pool": 2,
"old_pools": []
}

The directory containing the .snap dir is /AAA/BBB/CCC.
But the snapshot is in /AAA/BBB/CCC/.snap/$SNAPDIR.

For the client this might be only visible from a subdir depending on the subdir 
mounted.
In our case the client mounts with option name=AAA. The snapshot is visible as
/mnt/point/BBB/CCC/.snap/$SNAPDIR

The name of ".snap" depends on a mount option. This the metadata pool can't 
know. The name of the snapshot directory itself is not given.
You can only find directories where snapshots are in, not the actual snapdirs.

So you get the same information from find -inum. And it's simpler to use:

$ find /mnt/point/ -inum  -print -quit
/mnt/point/BBB/CCC

and then:
ls -1 /mnt/point/BBB/CCC/.snap


Best regards,
Lars

Wed, 18 Dec 2019 09:42:39 +
Frank Schilder  ==> Lars Täuber  :
> I found it, should have taken a note:
> 
> Command: rados -p  getxattr . 
> parent | ceph-dencoder type inode_backtrace_t import - decode dump_json
> 
> Note:  is hex encoded, use 'printf "%x\n" INUM' to 
> convert from the decimal numbers obtained with dump snaps.
> 
> Explanation: 
> 
> 
> I don't have snapshots enabled and cannot test. Could you confirm to the 
> mailing list that the above procedure will provide the path information all 
> the way down to the actual snap dir, that is, contains entries ".snap" and 
> "" at the top?
> 
> Best regards,
> 
> =
> Frank Schilder
> AIT Risø Campus
> Bygning 109, rum S14
> 
> 
> From: Lars Täuber 
> Sent: 18 December 2019 09:24:02
> To: Frank Schilder
> Cc: Marc Roos; ceph-users
> Subject: Re: [ceph-users] Re: list CephFS snapshots
> 
> Hi Frank,
> 
> thanks for you hint. The find for the inode is really fast. At least fast 
> enough for me:
> $ time find /mnt/point -inum 1093514215110 -print -quit
> real0m3,009s
> user0m0,037s
> sys 0m0,032s
> 
> Cheers,
> Lars
> 
> 
> Tue, 17 Dec 2019 15:08:06 +
> Frank Schilder  ==> Marc Roos , 
> taeuber  :
> > I think you can do a find for the inode (-inum n). At last I hope you can.
> >
> > However, I vaguely remember that there was a thread where someone gave a 
> > really nice MDS command for finding the path to an inode in no time.
> >
> > Best regards,
> >
> > =
> > Frank Schilder
> > AIT Risø Campus
> > Bygning 109, rum S14
> >
> > 
> > From: Marc Roos 
> > Sent: 17 December 2019 14:19:54
> > To: Frank Schilder; taeuber
> > Cc: ceph-users
> > Subject: RE: [ceph-users] Re: list CephFS snapshots
> >
> > Thanks, Good tip! If I do not know where I created these, is there a way
> > to get their location in the filesystem? Or maybe a command that deletes
> > by snapid?
> >
> >
> > {
> > "snapid": 54,
> > "ino": 1099519875627,
> > "stamp": "2017-09-13 21:21:35.769863",
> > "name": "snap-20170913"
> > },
> > {
> > "snapid": 153485,
> > "ino": 1099519910289,
> > "stamp": "2019-10-06 03:18:03.933510",
> > "name": "snap-6"
> > },
> > {
> > "snapid": 153489,
> > "ino": 1099519910289,
> > "stamp": "2019-10-07 03:21:03.218324",
> > "name": "snap-7"
> > },
> >
> >
> >
> > -Original Message-
> > Cc: ceph-users@ceph.io
> > Subject: [ceph-users] Re: list CephFS snapshots
> >
> > Have you tried "ceph daemon mds.NAME dump snaps" (available since
> > mimic)?
> >
> > =
> > Frank Schilder
> > AIT Risø Campus
> > Bygning 109, rum S14
> >
> > 
> > From: Lars Täuber 
> > Sent: 17 December 2019 12:32:34
> > To: Stephan Mueller
> > Cc: ceph-users@ceph.io
> > Subject: [ceph-users] Re: list CephFS snapshots
> >
> > Hi Michael,
> >
> > thanks for your gist.
> > This is at least a way to do it. But there are many directories in our
> > cluster.
> > The "find $1 -type d" lasts for about 90 minutes to find all 2.6 million
> > directories.
> >
> > Is there another (faster) way e.g. via mds?
> >
> > Cheers,
> > Lars
> >
> >
> > Mon, 16 Dec 2019 17:03:41 +
> > Stephan Mueller  ==> "taeu...@bbaw.de"
> > , "ceph-users@ceph.io"  :  
> > > Hi La

[ceph-users] Re: list CephFS snapshots

[Marc Roos]

 
This is working!


rados -p fs_meta getxattr $(printf "%x" 1099519896355). parent | 
ceph-dencoder type inode_backtrace_t import - decode dump_json


{
"ino": 1099519896355,
"ancestors": [
{
"dirino": 1099519874624,
"dname": "",
"version": 3299
},
{
"dirino": 1,
"dname": "xxx",
"version": 1424947
}
],
"pool": 19,
"old_pools": []
}

-Original Message-
From: Frank Schilder [mailto:fr...@dtu.dk] 
Sent: 18 December 2019 10:43
To: Lars Täuber
Cc: Marc Roos; ceph-users
Subject: Re: [ceph-users] Re: list CephFS snapshots

I found it, should have taken a note:

Command: rados -p  getxattr . parent | ceph-dencoder type inode_backtrace_t import - 
decode dump_json

Note:  is hex encoded, use 'printf "%x\n" INUM' to 
convert from the decimal numbers obtained with dump snaps.

Explanation: 


I don't have snapshots enabled and cannot test. Could you confirm to the 
mailing list that the above procedure will provide the path information 
all the way down to the actual snap dir, that is, contains entries 
".snap" and "" at the top?

Best regards,

=
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14


From: Lars Täuber 
Sent: 18 December 2019 09:24:02
To: Frank Schilder
Cc: Marc Roos; ceph-users
Subject: Re: [ceph-users] Re: list CephFS snapshots

Hi Frank,

thanks for you hint. The find for the inode is really fast. At least 
fast enough for me:
$ time find /mnt/point -inum 1093514215110 -print -quit
real0m3,009s
user0m0,037s
sys 0m0,032s

Cheers,
Lars


Tue, 17 Dec 2019 15:08:06 +
Frank Schilder  ==> Marc Roos , 
taeuber  :
> I think you can do a find for the inode (-inum n). At last I hope you 
can.
>
> However, I vaguely remember that there was a thread where someone gave 
a really nice MDS command for finding the path to an inode in no time.
>
> Best regards,
>
> =
> Frank Schilder
> AIT Risø Campus
> Bygning 109, rum S14
>
> 
> From: Marc Roos 
> Sent: 17 December 2019 14:19:54
> To: Frank Schilder; taeuber
> Cc: ceph-users
> Subject: RE: [ceph-users] Re: list CephFS snapshots
>
> Thanks, Good tip! If I do not know where I created these, is there a 
way
> to get their location in the filesystem? Or maybe a command that 
deletes
> by snapid?
>
>
> {
> "snapid": 54,
> "ino": 1099519875627,
> "stamp": "2017-09-13 21:21:35.769863",
> "name": "snap-20170913"
> },
> {
> "snapid": 153485,
> "ino": 1099519910289,
> "stamp": "2019-10-06 03:18:03.933510",
> "name": "snap-6"
> },
> {
> "snapid": 153489,
> "ino": 1099519910289,
> "stamp": "2019-10-07 03:21:03.218324",
> "name": "snap-7"
> },
>
>
>
> -Original Message-
> Cc: ceph-users@ceph.io
> Subject: [ceph-users] Re: list CephFS snapshots
>
> Have you tried "ceph daemon mds.NAME dump snaps" (available since
> mimic)?
>
> =
> Frank Schilder
> AIT Risø Campus
> Bygning 109, rum S14
>
> 
> From: Lars Täuber 
> Sent: 17 December 2019 12:32:34
> To: Stephan Mueller
> Cc: ceph-users@ceph.io
> Subject: [ceph-users] Re: list CephFS snapshots
>
> Hi Michael,
>
> thanks for your gist.
> This is at least a way to do it. But there are many directories in our
> cluster.
> The "find $1 -type d" lasts for about 90 minutes to find all 2.6 
million
> directories.
>
> Is there another (faster) way e.g. via mds?
>
> Cheers,
> Lars
>
>
> Mon, 16 Dec 2019 17:03:41 +
> Stephan Mueller  ==> "taeu...@bbaw.de"
> , "ceph-users@ceph.io"  :
> > Hi Lars,
> >
> > > Is there a mean to list all snapshots existing in a (subdir of)
> > > Cephfs?
> > > I can't use the find dommand to look for the ".snap" dirs.
> >
> > You can, but you can't search for the '.snap' directories, you have 
to
>
> > append them to the directory like `find $cephFsDir/.snap` but I it's
> > better to use `ls` instead, to list all snapshots.
> >
> > >
> > > I'd like to remove certain (or all) snapshots within a CephFS. But
> > > how do I find them?
> > >
> >
> > I just created a gist for you that can do that:
> > https://gist.github.com/Devp00l/2473f5953d578f440fc71b3d602a9c23
> >
> > As you can see in the script, snapshots starting with an underscore
> > are filtered out as these directories belong to snapshots that were
> > created in upper directories and these underscore snapshots can't be
> > used for deletion.
> >
> > The deletion of a snapshot can be done by calling `rmdir`.
> >
> > But if you really want to manage CephFS snapshots easily take a look
> > at the dashboard, as we have integrated the snapshot and quota
> > 

[ceph-users] Use Wireshark to analysis ceph network package

[Xu Chen]

Hi guys, I want to use tcpdump and Wireshark to capture and analysis
packages between clients and ceph cluster. But the protocol only shows tcp,
no ceph, so I can not read the data between client and cluster. The
wireshark version is 3.07.

Hope your help.
Thank you.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] re-balancing resulting in unexpected availability issues

[steve . nolen]

Hi!

We've found ourselves in state with our ceph cluster that we haven't seen 
before, and are looking for a bit of expertise to chime in. We're running a 
(potentially unusually laid out) moderately large luminous-based ceph cluster 
in a public cloud, with 234*8TB OSDs, with a single osd per cloud instance. 
Here's a snippet of our ceph status:

services:
mon: 3 daemons, quorum ceph-mon1,ceph-mon2,ceph-mon4
mgr: ceph-mon2(active), standbys: ceph-mon1, ceph-mon4
osd: 234 osds: 234 up, 231 in

data:
pools: 5 pools, 7968 pgs
objects: 136.35M objects, 382TiB
usage: 1.09PiB used, 713TiB / 1.79PiB avail
pgs: 7924 active+clean
44 active+clean+scrubbing+deep

Our layout is spread across 6 availability zones (with the majority of osds in 
three, us-east-1a,us-east-1b and us-east-1e). We've recently decided that a 
spread across six azs is unnecessary and potentially a detriment to 
performance, so we are working towards shifting our workload out of 3 of the 6 
azs, so that we are evenly placed in 3 azs.

Relevant Recent events:
1. We expanded by 24 osds to handle additional capacity needs as well as to 
provide the capacity necessary to remove all osds in us-east-1f.
2. After the re-balance related to #1 finished we expanded by an additional 12 
osds as a follow-on for the #1 change.
3. On the same day as #2, we also issued `ceph crush move` commands to move the 
location of oldest 20 osds which had previously not been configured with a 
"datacenter" bucket denoting their availability zone.

The re-balancing related to #3 caused quite a change in our cluster, resulting 
in hours with degraded pgs, and waves of "slow requests" from many of the osds 
as data shifted. Three of the osds which had their location moved are also 
wildly more full than the other osds (being in a nearfull, >85% utilized state 
where the other 231 osds are in the range of 58-63% utilized). A `ceph balancer 
optimize` plan shows no changes to be made by the balancer to rectify this. 
Because of their nearfull status, we have marked those three osds as "out". 
During the resulting re-balancing we experienced more "slow requests" piling 
up, with some pgs dipping into a peering or activating state (which obviously 
causes some user-visible trouble).

Where we are headed:
* We have yet to set the crush map to use a replicated_datacenter rule now that 
our osds conform to these buckets.
* We need to take action to remove/replace the three osds which were 
over-utilized.
* We need to remove the us-east-1f osds that provoked the event in #1 (of which 
there are 16).

What I'm hoping for a bit of direction on:
* A recommendation on the order of these operations that would result in the 
least user-visible time.
* Some hints or thoughts on why we are seeing flapping pg availability since 
the change in #3.
* General thoughts on our layout. It has been rather useful for unbounded rapid 
growth but it seems very likely the sheer count of osds/instances is optimal at 
this point.

Obviously I'm happy to provide any additional information that might be helpful 
that I've overlooked. Thanks so much for looking!
Steve
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] High CPU usage by ceph-mgr in 14.2.5

[Bryan Stillwell]

After upgrading one of our clusters from Nautilus 14.2.2 to Nautilus 14.2.5 I'm 
seeing 100% CPU usage by a single ceph-mgr thread (found using 'top -H').  
Attaching to the thread with strace shows a lot of mmap and munmap calls.  
Here's the distribution after watching it for a few minutes:

48.73% - mmap
49.48% - munmap
1.75% - futex
0.05% - madvise

I've upgraded 3 other clusters so far (120 OSDs, 30 OSDs, 200 OSDs), but this 
is the only one which has seen the problem (355 OSDs).  Perhaps it has 
something to do with its size?

I was suspecting it might have to do with one of the modules misbehaving, so I 
disabled all of them:

# ceph mgr module ls | jq -r '.enabled_modules'
[]

But that didn't help (I restarted the mgrs after disabling the modules too).

I also tried setting debug_mgr and debug_mgrc to 20, but nothing popped out at 
me as being the cause of the problem.

It only seems to affect the active mgr.  If I stop the active mgr the problem 
moves to one of the other mgrs.

Any guesses or tips on what next steps I should take to figure out what's going 
on?

Thanks,
Bryan
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: High CPU usage by ceph-mgr in 14.2.5

[Sage Weil]

On Wed, 18 Dec 2019, Bryan Stillwell wrote:
> After upgrading one of our clusters from Nautilus 14.2.2 to Nautilus 14.2.5 
> I'm seeing 100% CPU usage by a single ceph-mgr thread (found using 'top -H'). 
>  Attaching to the thread with strace shows a lot of mmap and munmap calls.  
> Here's the distribution after watching it for a few minutes:
> 
> 48.73% - mmap
> 49.48% - munmap
> 1.75% - futex
> 0.05% - madvise
> 
> I've upgraded 3 other clusters so far (120 OSDs, 30 OSDs, 200 OSDs), but this 
> is the only one which has seen the problem (355 OSDs).  Perhaps it has 
> something to do with its size?
> 
> I was suspecting it might have to do with one of the modules misbehaving, so 
> I disabled all of them:
> 
> # ceph mgr module ls | jq -r '.enabled_modules'
> []
> 
> But that didn't help (I restarted the mgrs after disabling the modules too).
> 
> I also tried setting debug_mgr and debug_mgrc to 20, but nothing popped out 
> at me as being the cause of the problem.
> 
> It only seems to affect the active mgr.  If I stop the active mgr the problem 
> moves to one of the other mgrs.
> 
> Any guesses or tips on what next steps I should take to figure out what's 
> going on?

What are the balancer modes on the affected and unaffected cluster(s)?

sage
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: High CPU usage by ceph-mgr in 14.2.5

[eric]

Hey,

That sounds very similar to what I described there: 
https://tracker.ceph.com/issues/43364

Best,
Eric
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: High CPU usage by ceph-mgr in 14.2.5

[Bryan Stillwell]

On Dec 18, 2019, at 1:48 PM, e...@lapsus.org wrote:
> 
> That sounds very similar to what I described there: 
> https://tracker.ceph.com/issues/43364

I would agree that they're quite similar if not the same thing!  Now that you 
mention it I see the thread is named mgr-fin in 'top -H' as well.  I don't have 
really high queue length like you do though, but it does bounce around between 
0 and 200, so maybe I'm right at the edge for cluster size before it starts 
growing out of control?

Thanks,
Bryan
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: High CPU usage by ceph-mgr in 14.2.5

[Bryan Stillwell]

On Dec 18, 2019, at 11:58 AM, Sage Weil 
mailto:s...@newdream.net>> wrote:

On Wed, 18 Dec 2019, Bryan Stillwell wrote:
After upgrading one of our clusters from Nautilus 14.2.2 to Nautilus 14.2.5 I'm 
seeing 100% CPU usage by a single ceph-mgr thread (found using 'top -H').  
Attaching to the thread with strace shows a lot of mmap and munmap calls.  
Here's the distribution after watching it for a few minutes:

48.73% - mmap
49.48% - munmap
1.75% - futex
0.05% - madvise

I've upgraded 3 other clusters so far (120 OSDs, 30 OSDs, 200 OSDs), but this 
is the only one which has seen the problem (355 OSDs). Perhaps it has something 
to do with its size?

I was suspecting it might have to do with one of the modules misbehaving, so I 
disabled all of them:

# ceph mgr module ls | jq -r '.enabled_modules'
[]

But that didn't help (I restarted the mgrs after disabling the modules too).

I also tried setting debug_mgr and debug_mgrc to 20, but nothing popped out at 
me as being the cause of the problem.

It only seems to affect the active mgr.  If I stop the active mgr the problem 
moves to one of the other mgrs.

Any guesses or tips on what next steps I should take to figure out what's going 
on?

What are the balancer modes on the affected and unaffected cluster(s)?

Affected cluster has a balancer mode of "none".

The other three are "upmap", "none", and "upmap".

I don't know if you saw in ceph-users, but this bug report seems to point at 
the finisher-Mgr thread:

https://tracker.ceph.com/issues/43364

Thanks,
Bryan
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: High CPU usage by ceph-mgr in 14.2.5

[Paul Mezzanini]

Just wanted to say that we are seeing the same thing on our large cluster.   It 
manifested mainly in the from of Prometheus stats being totally broken (they 
take too long to return if at all so the requesting program just gives up)


--
Paul Mezzanini
Sr Systems Administrator / Engineer, Research Computing
Information & Technology Services
Finance & Administration
Rochester Institute of Technology
o:(585) 475-3245 | pfm...@rit.edu

Sent from my phone. Please excuse any brevity or typoos.

CONFIDENTIALITY NOTE: The information transmitted, including attachments, is
intended only for the person(s) or entity to which it is addressed and may
contain confidential and/or privileged material. Any review, retransmission,
dissemination or other use of, or taking of any action in reliance upon this
information by persons or entities other than the intended recipient is
prohibited. If you received this in error, please contact the sender and
destroy any copies of this information.



From: Bryan Stillwell 
Sent: Wednesday, December 18, 2019 4:44:45 PM
To: Sage Weil 
Cc: ceph-users ; d...@ceph.io 
Subject: [ceph-users] Re: High CPU usage by ceph-mgr in 14.2.5

On Dec 18, 2019, at 11:58 AM, Sage Weil 
mailto:s...@newdream.net>> wrote:

On Wed, 18 Dec 2019, Bryan Stillwell wrote:
After upgrading one of our clusters from Nautilus 14.2.2 to Nautilus 14.2.5 I'm 
seeing 100% CPU usage by a single ceph-mgr thread (found using 'top -H').  
Attaching to the thread with strace shows a lot of mmap and munmap calls.  
Here's the distribution after watching it for a few minutes:

48.73% - mmap
49.48% - munmap
1.75% - futex
0.05% - madvise

I've upgraded 3 other clusters so far (120 OSDs, 30 OSDs, 200 OSDs), but this 
is the only one which has seen the problem (355 OSDs). Perhaps it has something 
to do with its size?

I was suspecting it might have to do with one of the modules misbehaving, so I 
disabled all of them:

# ceph mgr module ls | jq -r '.enabled_modules'
[]

But that didn't help (I restarted the mgrs after disabling the modules too).

I also tried setting debug_mgr and debug_mgrc to 20, but nothing popped out at 
me as being the cause of the problem.

It only seems to affect the active mgr.  If I stop the active mgr the problem 
moves to one of the other mgrs.

Any guesses or tips on what next steps I should take to figure out what's going 
on?

What are the balancer modes on the affected and unaffected cluster(s)?

Affected cluster has a balancer mode of "none".

The other three are "upmap", "none", and "upmap".

I don't know if you saw in ceph-users, but this bug report seems to point at 
the finisher-Mgr thread:

https://tracker.ceph.com/issues/43364

Thanks,
Bryan
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: High CPU usage by ceph-mgr in 14.2.5

[Bryan Stillwell]

That's how we noticed it too.  Our graphs went silent after the upgrade 
completed.  Is your large cluster over 350 OSDs?

Bryan

On Dec 18, 2019, at 2:59 PM, Paul Mezzanini 
mailto:pfm...@rit.edu>> wrote:

Notice: This email is from an external sender.

Just wanted to say that we are seeing the same thing on our large cluster.   It 
manifested mainly in the from of Prometheus stats being totally broken (they 
take too long to return if at all so the requesting program just gives up)


--
Paul Mezzanini
Sr Systems Administrator / Engineer, Research Computing
Information & Technology Services
Finance & Administration
Rochester Institute of Technology
o:(585) 475-3245 | pfm...@rit.edu

Sent from my phone. Please excuse any brevity or typoos.

CONFIDENTIALITY NOTE: The information transmitted, including attachments, is
intended only for the person(s) or entity to which it is addressed and may
contain confidential and/or privileged material. Any review, retransmission,
dissemination or other use of, or taking of any action in reliance upon this
information by persons or entities other than the intended recipient is
prohibited. If you received this in error, please contact the sender and
destroy any copies of this information.



From: Bryan Stillwell mailto:bstillw...@godaddy.com>>
Sent: Wednesday, December 18, 2019 4:44:45 PM
To: Sage Weil mailto:s...@newdream.net>>
Cc: ceph-users mailto:ceph-users@ceph.io>>; 
d...@ceph.io mailto:d...@ceph.io>>
Subject: [ceph-users] Re: High CPU usage by ceph-mgr in 14.2.5

On Dec 18, 2019, at 11:58 AM, Sage Weil 
mailto:s...@newdream.net>> wrote:

On Wed, 18 Dec 2019, Bryan Stillwell wrote:
After upgrading one of our clusters from Nautilus 14.2.2 to Nautilus 14.2.5 I'm 
seeing 100% CPU usage by a single ceph-mgr thread (found using 'top -H').  
Attaching to the thread with strace shows a lot of mmap and munmap calls.  
Here's the distribution after watching it for a few minutes:

48.73% - mmap
49.48% - munmap
1.75% - futex
0.05% - madvise

I've upgraded 3 other clusters so far (120 OSDs, 30 OSDs, 200 OSDs), but this 
is the only one which has seen the problem (355 OSDs). Perhaps it has something 
to do with its size?

I was suspecting it might have to do with one of the modules misbehaving, so I 
disabled all of them:

# ceph mgr module ls | jq -r '.enabled_modules'
[]

But that didn't help (I restarted the mgrs after disabling the modules too).

I also tried setting debug_mgr and debug_mgrc to 20, but nothing popped out at 
me as being the cause of the problem.

It only seems to affect the active mgr.  If I stop the active mgr the problem 
moves to one of the other mgrs.

Any guesses or tips on what next steps I should take to figure out what's going 
on?

What are the balancer modes on the affected and unaffected cluster(s)?

Affected cluster has a balancer mode of "none".

The other three are "upmap", "none", and "upmap".

I don't know if you saw in ceph-users, but this bug report seems to point at 
the finisher-Mgr thread:

https://tracker.ceph.com/issues/43364

Thanks,
Bryan

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: High CPU usage by ceph-mgr in 14.2.5

[Andras Pataki]

We are also running into this issue on one of our clusters - balancer 
mode upmap, about 950 OSDs.


Andras


On 12/18/19 4:44 PM, Bryan Stillwell wrote:
On Dec 18, 2019, at 11:58 AM, Sage Weil > wrote:


On Wed, 18 Dec 2019, Bryan Stillwell wrote:
After upgrading one of our clusters from Nautilus 14.2.2 to Nautilus 
14.2.5 I'm seeing 100% CPU usage by a single ceph-mgr thread (found 
using 'top -H').  Attaching to the thread with strace shows a lot of 
mmap and munmap calls.  Here's the distribution after watching it 
for a few minutes:


48.73% - mmap
49.48% - munmap
1.75% - futex
0.05% - madvise

I've upgraded 3 other clusters so far (120 OSDs, 30 OSDs, 200 OSDs), 
but this is the only one which has seen the problem (355 OSDs). 
Perhaps it has something to do with its size?


I was suspecting it might have to do with one of the modules 
misbehaving, so I disabled all of them:


# ceph mgr module ls | jq -r '.enabled_modules'
[]

But that didn't help (I restarted the mgrs after disabling the 
modules too).


I also tried setting debug_mgr and debug_mgrc to 20, but nothing 
popped out at me as being the cause of the problem.


It only seems to affect the active mgr.  If I stop the active mgr 
the problem moves to one of the other mgrs.


Any guesses or tips on what next steps I should take to figure out 
what's going on?


What are the balancer modes on the affected and unaffected cluster(s)?


Affected cluster has a balancer mode of "none".

The other three are "upmap", "none", and "upmap".

I don't know if you saw in ceph-users, but this bug report seems to 
point at the finisher-Mgr thread:


https://tracker.ceph.com/issues/43364

Thanks,
Bryan

___
Dev mailing list -- d...@ceph.io
To unsubscribe send an email to dev-le...@ceph.io


___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: High CPU usage by ceph-mgr in 14.2.5

[Paul Mezzanini]

>From memory we are in the 700s.


--
Paul Mezzanini
Sr Systems Administrator / Engineer, Research Computing
Information & Technology Services
Finance & Administration
Rochester Institute of Technology
o:(585) 475-3245 | pfm...@rit.edu

Sent from my phone. Please excuse any brevity or typoos.

CONFIDENTIALITY NOTE: The information transmitted, including attachments, is
intended only for the person(s) or entity to which it is addressed and may
contain confidential and/or privileged material. Any review, retransmission,
dissemination or other use of, or taking of any action in reliance upon this
information by persons or entities other than the intended recipient is
prohibited. If you received this in error, please contact the sender and
destroy any copies of this information.



From: Bryan Stillwell 
Sent: Wednesday, December 18, 2019 5:40:22 PM
To: Paul Mezzanini 
Cc: ceph-users ; d...@ceph.io 
Subject: Re: High CPU usage by ceph-mgr in 14.2.5

That's how we noticed it too.  Our graphs went silent after the upgrade 
completed.  Is your large cluster over 350 OSDs?

Bryan

On Dec 18, 2019, at 2:59 PM, Paul Mezzanini 
mailto:pfm...@rit.edu>> wrote:

Notice: This email is from an external sender.

Just wanted to say that we are seeing the same thing on our large cluster.   It 
manifested mainly in the from of Prometheus stats being totally broken (they 
take too long to return if at all so the requesting program just gives up)


--
Paul Mezzanini
Sr Systems Administrator / Engineer, Research Computing
Information & Technology Services
Finance & Administration
Rochester Institute of Technology
o:(585) 475-3245 | pfm...@rit.edu

Sent from my phone. Please excuse any brevity or typoos.

CONFIDENTIALITY NOTE: The information transmitted, including attachments, is
intended only for the person(s) or entity to which it is addressed and may
contain confidential and/or privileged material. Any review, retransmission,
dissemination or other use of, or taking of any action in reliance upon this
information by persons or entities other than the intended recipient is
prohibited. If you received this in error, please contact the sender and
destroy any copies of this information.



From: Bryan Stillwell mailto:bstillw...@godaddy.com>>
Sent: Wednesday, December 18, 2019 4:44:45 PM
To: Sage Weil mailto:s...@newdream.net>>
Cc: ceph-users mailto:ceph-users@ceph.io>>; 
d...@ceph.io mailto:d...@ceph.io>>
Subject: [ceph-users] Re: High CPU usage by ceph-mgr in 14.2.5

On Dec 18, 2019, at 11:58 AM, Sage Weil 
mailto:s...@newdream.net>> wrote:

On Wed, 18 Dec 2019, Bryan Stillwell wrote:
After upgrading one of our clusters from Nautilus 14.2.2 to Nautilus 14.2.5 I'm 
seeing 100% CPU usage by a single ceph-mgr thread (found using 'top -H').  
Attaching to the thread with strace shows a lot of mmap and munmap calls.  
Here's the distribution after watching it for a few minutes:

48.73% - mmap
49.48% - munmap
1.75% - futex
0.05% - madvise

I've upgraded 3 other clusters so far (120 OSDs, 30 OSDs, 200 OSDs), but this 
is the only one which has seen the problem (355 OSDs). Perhaps it has something 
to do with its size?

I was suspecting it might have to do with one of the modules misbehaving, so I 
disabled all of them:

# ceph mgr module ls | jq -r '.enabled_modules'
[]

But that didn't help (I restarted the mgrs after disabling the modules too).

I also tried setting debug_mgr and debug_mgrc to 20, but nothing popped out at 
me as being the cause of the problem.

It only seems to affect the active mgr.  If I stop the active mgr the problem 
moves to one of the other mgrs.

Any guesses or tips on what next steps I should take to figure out what's going 
on?

What are the balancer modes on the affected and unaffected cluster(s)?

Affected cluster has a balancer mode of "none".

The other three are "upmap", "none", and "upmap".

I don't know if you saw in ceph-users, but this bug report seems to point at 
the finisher-Mgr thread:

https://tracker.ceph.com/issues/43364

Thanks,
Bryan

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io