Send ceph-users mailing list submissions to
ceph-users@ceph.io
To subscribe or unsubscribe via email, send a message with subject or
body 'help' to
ceph-users-requ...@ceph.io
You can reach the person managing the list at
ceph-users-ow...@ceph.io
When replying, please edit your Subject line so it is more specific
than "Re: Contents of ceph-users digest..."
Today's Topics:
1. Re: Removing pool in nautilus is incredibly slow
(Francois Legrand)
2. Re: Bench on specific OSD (Marc Roos)
3. Re: Removing pool in nautilus is incredibly slow
(Wout van Heeswijk)
4. Lifecycle message on logs (Marcelo Miziara)
5. Re: Feedback of the used configuration (Simon Sutter)
6. Re: Removing pool in nautilus is incredibly slow
(Francois Legrand)
7. Re: Removing pool in nautilus is incredibly slow (Eugen Block)
----------------------------------------------------------------------
Date: Thu, 25 Jun 2020 07:57:48 -0000
From: "Francois Legrand" <f...@lpnhe.in2p3.fr>
Subject: [ceph-users] Re: Removing pool in nautilus is incredibly slow
To: ceph-users@ceph.io
Message-ID: <159307186843.20.7354239610800797635@mailman-web>
Content-Type: text/plain; charset="utf-8"
Does someone have an idea ?
F.
------------------------------
Date: Thu, 25 Jun 2020 11:36:24 +0200
From: "Marc Roos" <m.r...@f1-outsourcing.eu>
Subject: [ceph-users] Re: Bench on specific OSD
To: ceph-users <ceph-users@ceph.io>, seenafallah
<seenafal...@gmail.com>
Message-ID: <"H0000071001729d1.1593077784.sx.f1-outsourcing.eu*"@MHS>
Content-Type: text/plain; charset="US-ASCII"
What is wrong with just doing multiple tests and group in your charts
osd's by host?
-----Original Message-----
To: ceph-users
Subject: [ceph-users] Bench on specific OSD
Hi all.
Is there anyway to completely health check one OSD host or instance?
For example rados bech just on that OSD or do some checks for disk and
front and back netowrk?
Thanks.
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an
email to ceph-users-le...@ceph.io
------------------------------
Date: Thu, 25 Jun 2020 14:26:59 +0200
From: Wout van Heeswijk <w...@42on.com>
Subject: [ceph-users] Re: Removing pool in nautilus is incredibly slow
To: f...@lpnhe.in2p3.fr
Cc: ceph-users@ceph.io
Message-ID: <38a5d557-7797-a68a-1b67-4c8f0e1ec...@42on.com>
Content-Type: text/plain; charset=utf-8; format=flowed
Hi Francois,
Have you already looked at the option "osd_delete_sleep"? It will not
speed up the process but I will give you some control over your cluster
performance.
Something like:
ceph tell osd.\* injectargs '--osd_delete_sleep1'
kind regards,
Wout
42on
On 25-06-2020 09:57, Francois Legrand wrote:
Does someone have an idea ?
F.
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
------------------------------
Date: Thu, 25 Jun 2020 09:53:09 -0300
From: Marcelo Miziara <raxi...@gmail.com>
Subject: [ceph-users] Lifecycle message on logs
To: ceph-users@ceph.io
Message-ID:
<CAGxsqB296eczKux19OKObkimXg3PZ_UDEWVTTqEnrAf=agm...@mail.gmail.com>
Content-Type: text/plain; charset="UTF-8"
Hello...it's the first time I need to use the lifecycle, and I created a
bucket and set it to expire in one day with s3cmd:
s3cmd expire --expiry-days=1 s3://bucket
The rgw_lifecycle_work_time is set to the default values(00:00-06:00). But
I noticed in the rgw logs a lot of messages like:
2020-06-16 00:00:00.311369 7fe2cac87700 0 RGWLC::process() failed to get
obj entry lc.8
2020-06-16 00:00:00.311623 7fe2c8c83700 0 RGWLC::process() failed to get
obj entry lc.16
2020-06-16 00:00:00.311862 7fe2c6c7f700 0 RGWLC::process() failed to get
obj entry lc.4
2020-06-16 00:00:00.319424 7fe2cac87700 0 RGWLC::process() failed to get
obj entry lc.10
2020-06-16 00:00:00.319647 7fe2c8c83700 0 RGWLC::process() failed to get
obj entry lc.18
2020-06-16 00:00:00.320682 7fe2c6c7f700 0 RGWLC::process() failed to get
obj entry lc.16
2020-06-16 00:00:00.327770 7fe2cac87700 0 RGWLC::process() failed to get
obj entry lc.6
2020-06-16 00:00:00.328941 7fe2c8c83700 0 RGWLC::process() failed to get
obj entry lc.17
2020-06-16 00:00:00.332463 7fe2c6c7f700 0 RGWLC::process() failed to get
obj entry lc.20
2020-06-16 00:00:00.336788 7fe2cac87700 0 RGWLC::process() failed to get
obj entry lc.1
2020-06-16 00:00:00.336924 7fe2c8c83700 0 RGWLC::process() failed to get
obj entry lc.24
2020-06-16 00:00:00.340915 7fe2c6c7f700 0 RGWLC::process() failed to get
obj entry lc.2
The object was deleted, but these messages keep appearing.
Is it safe to ignore them?
For the records, i'm using redhat luminous 12.2.12
Thanks, Marcelo.
------------------------------
Date: Thu, 25 Jun 2020 13:19:38 +0000
From: Simon Sutter <ssut...@hosttech.ch>
Subject: [ceph-users] Re: Feedback of the used configuration
To: Paul Emmerich <paul.emmer...@croit.io>
Cc: "ceph-users@ceph.io" <ceph-users@ceph.io>
Message-ID: <f6e11ee189a74c35815c495d64cbe...@hosttech.ch>
Content-Type: text/plain; charset="utf-8"
Hello Paul,
Thanks for the Answer.
I took a look at the subvolumes, but they are a bit odd in my opinion.
If I create one with a subvolume-group, the folder structure will look like
this:
/cephfs/volumes/group-name/subvolume-name/random-uuid/
And I have to issue two commands, first set the group and then set the volume
name, but why so complicated?
Wouldn't it be easier to just make subvolumes anywhere inside the cephfs?
I can see the intended use for groups, but if I want to publish a pool in some
different directory that's not possible (except for setfattr).
Without first creating subvolume-groups, the orchestrator creates subvolumes in
the /cephfs/volumes/_nogroup/subvolume-name/randmon-uuid/ folder.
And the more important question is, why is there a new folder with a random
uuid inside the subvolume?
I try to understand the points the devs had, when they developed this, but for
me, this is something I have to explain to some devs in our team and at the
moment I can't.
It is indeed easier to deploy but comes with much less flexibility.
Maybe something to write in the tracker about?
Thanks in advance,
Simon
Von: Paul Emmerich [mailto:paul.emmer...@croit.io]
Gesendet: Mittwoch, 24. Juni 2020 17:35
An: Simon Sutter <ssut...@hosttech.ch>
Cc: ceph-users@ceph.io
Betreff: Re: [ceph-users] Feedback of the used configuration
Have a look at cephfs subvolumes:
https://docs.ceph.com/docs/master/cephfs/fs-volumes/#fs-subvolumes
They are internally just directories with quota/pool placement layout/namespace
with some mgr magic to make it easier than doing that all by hand
Paul
--
Paul Emmerich
Looking for help with your Ceph cluster? Contact us at https://croit.io
croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io<http://www.croit.io>
Tel: +49 89 1896585 90
On Wed, Jun 24, 2020 at 4:38 PM Simon Sutter
<ssut...@hosttech.ch<mailto:ssut...@hosttech.ch>> wrote:
Hello,
After two months of the "ceph try and error game", I finally managed to get an
Octopuss cluster up and running.
The unconventional thing about it is, it's just for hot backups, no virtual
machines on there.
All the nodes are without any caching ssd's, just plain hdd's.
At the moment there are eight of them with a total of 50TB. We are planning to
go up to 25 and bigger disks so we end on 300TB-400TB
I decided to go with cephfs, because I don't have any experience in things like
S3 and I need to read the same file system from more than one client.
I made one cephfs with a replicated pool.
On there I added erasure-coded pools to save some Storage.
To add those pools, I did it with the setfattr command like this:
setfattr -n ceph.dir.layout.pool -v ec_data_server1 /cephfs/nfs/server1
Some of our servers cannot use cephfs (old kernels, special OS's) so I have to
use nfs.
This is set up with the included ganesha-nfs.
Exported is the /cephfs/nfs folder and clients can mount folders below this.
There are two final questions:
- Was it right to go with the way of "mounting" pools with setfattr,
or should I have used multiple cephfs?
First I was thinking about using multiple cephfs but there are warnings
everywhere. The deeper I got in, the more it seems I would have been fine with
multiple cephfs.
- Is there a way I don't know, but it would be easier?
I still don't know much about Rest, S3, RBD etc... so there may be a better way
Other remarks are desired.
Thanks in advance,
Simon
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io>
To unsubscribe send an email to
ceph-users-le...@ceph.io<mailto:ceph-users-le...@ceph.io>
------------------------------
Date: Thu, 25 Jun 2020 16:22:12 +0200
From: Francois Legrand <f...@lpnhe.in2p3.fr>
Subject: [ceph-users] Re: Removing pool in nautilus is incredibly slow
To: Wout van Heeswijk <w...@42on.com>
Cc: ceph-users@ceph.io
Message-ID: <db34e791-1172-8b94-09e7-4ab5790d3...@lpnhe.in2p3.fr>
Content-Type: text/plain; charset=utf-8; format=flowed
Thanks for the hint.
I tryed but it doesn't seems to change anything...
Moreover, as the osds seems quite loaded I had regularly some osd marked
down which triggered some new peering and thus more load !!!
I set the osd no down flag, but I still have some osd reported (wrongly)
as down (and back up in the minute) which generate peering and
remapping. I don't really understand the action of no down parameter !
Is there a way to tell ceph not to peer immediately after an osd is
reported down (let say wait for 60s) ?
I am thinking about restarting all osd (or maybe the whole cluster) to
get osd_op_queue_cut_off changed to high and osd_op_thread_timeout to
something higher than 15 (but I don't think it will really improve the
situation).
F.
Le 25/06/2020 à 14:26, Wout van Heeswijk a écrit :
Hi Francois,
Have you already looked at the option "osd_delete_sleep"? It will not
speed up the process but I will give you some control over your
cluster performance.
Something like:
ceph tell osd.\* injectargs '--osd_delete_sleep1'
kind regards,
Wout
42on
On 25-06-2020 09:57, Francois Legrand wrote:
Does someone have an idea ?
F.
_______________________________________________
ceph-users mailing list --ceph-users@ceph.io
To unsubscribe send an email toceph-users-le...@ceph.io
------------------------------
Date: Thu, 25 Jun 2020 14:42:57 +0000
From: Eugen Block <ebl...@nde.ag>
Subject: [ceph-users] Re: Removing pool in nautilus is incredibly slow
To: ceph-users@ceph.io
Message-ID:
<20200625144257.horde.i8jpcddeor47wdoehqkq...@webmail.nde.ag>
Content-Type: text/plain; charset=utf-8; format=flowed; DelSp=Yes
I'm not sure if your OSDs have their rocksDB on faster devices, if not
it sounds a lot like rocksdb fragmentation [1] leading to a very high
load on the OSDs and occasionally crashing OSDs. If you don't plan to
delete so much data at once on a regular basis you could sit this one
out, but one solution is to re-create the OSDs with rocksDB/WAL on
faster devices.
[1] https://www.mail-archive.com/ceph-users@ceph.io/msg03160.html
Zitat von Francois Legrand <f...@lpnhe.in2p3.fr>:
Thanks for the hint.
I tryed but it doesn't seems to change anything...
Moreover, as the osds seems quite loaded I had regularly some osd
marked down which triggered some new peering and thus more load !!!
I set the osd no down flag, but I still have some osd reported
(wrongly) as down (and back up in the minute) which generate peering
and remapping. I don't really understand the action of no down
parameter !
Is there a way to tell ceph not to peer immediately after an osd is
reported down (let say wait for 60s) ?
I am thinking about restarting all osd (or maybe the whole cluster)
to get osd_op_queue_cut_off changed to high and
osd_op_thread_timeout to something higher than 15 (but I don't think
it will really improve the situation).
F.
Le 25/06/2020 à 14:26, Wout van Heeswijk a écrit :
Hi Francois,
Have you already looked at the option "osd_delete_sleep"? It will
not speed up the process but I will give you some control over your
cluster performance.
Something like:
ceph tell osd.\* injectargs '--osd_delete_sleep1'
kind regards,
Wout
42on
On 25-06-2020 09:57, Francois Legrand wrote:
Does someone have an idea ?
F.
_______________________________________________
ceph-users mailing list --ceph-users@ceph.io
To unsubscribe send an email toceph-users-le...@ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
------------------------------
Subject: Digest Footer
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
%(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s
------------------------------
End of ceph-users Digest, Vol 89, Issue 112
*******************************************
