Re: [ceph-users] pros/cons of multiple OSD's per host

On Tue, 22 Aug 2017 09:32:20 +0800 Nick Tan wrote: > On Mon, Aug 21, 2017 at 8:57 PM, David Turner wrote: > > > It is not recommended to get your cluster more than 70% full due to > > rebalancing and various other reasons. That would change your 12x 10TB > > disks in a host to only be 84TB if yo

Re: [ceph-users] pros/cons of multiple OSD's per host

On Mon, Aug 21, 2017 at 8:57 PM, David Turner wrote: > It is not recommended to get your cluster more than 70% full due to > rebalancing and various other reasons. That would change your 12x 10TB > disks in a host to only be 84TB if you filled your cluster to 70% full. I > still think that the mo

[ceph-users] RBD encryption options?

Are there any client-side options to encrypt an RBD device? Using latest luminous RC, on Ubuntu 16.04 and a 4.10 kernel I assumed adding client site encryption would be as simple as using luks/dm-crypt/cryptsetup after adding the RBD device to /etc/ceph/rbdmap and enabling the rbdmap service --

Re: [ceph-users] NVMe + SSD + HDD RBD Replicas with Bluestore...

Hello, Firstly, what David said. On Mon, 21 Aug 2017 20:25:07 + Xavier Trilla wrote: > Hi, > > I'm working into improving the costs of our actual ceph cluster. We actually > keep 3 x replicas, all of them in SSDs (That cluster hosts several hundred > VMs RBD disks) and lately I've been

[ceph-users] ceph-fuse hanging on df with ceph luminous >= 12.1.3

Hi, when trying to use df on a ceph-fuse mounted cephfs filesystem with ceph luminous >= 12.1.3 I'm having hangs with the following kind of messages in the logs: 2017-08-22 02:20:51.094704 7f80addb7700 0 client.174216 ms_handle_reset on 192.168.0.10:6789/0 The logs are only showing this

Re: [ceph-users] PG reported as inconsistent in status, but no inconsistencies visible to rados

Could you provide the output of 'ceph-bluestore-tool fsck' for one of these OSDs? On Tue, Aug 22, 2017 at 2:53 AM, Edward R Huyer wrote: > This is an odd one. My cluster is reporting an inconsistent pg in ceph > status and ceph health detail. However, rados list-inconsistent-obj and > rados lis

Re: [ceph-users] NVMe + SSD + HDD RBD Replicas with Bluestore...

I don't understand why min_size = 2 would kill latency times. Regardless of your min_size, a write to ceph does not ack until it completes to all copies. That means that even with min_size = 1 the write will not be successful until it's written to the NVME, the SSD, and the HDD (given your propos

[ceph-users] NVMe + SSD + HDD RBD Replicas with Bluestore...

Hi, I'm working into improving the costs of our actual ceph cluster. We actually keep 3 x replicas, all of them in SSDs (That cluster hosts several hundred VMs RBD disks) and lately I've been wondering if the following setup would make sense, in order to improve cost / performance. The ideal w

[ceph-users] [CEPH/SPDK]How accelerate Ceph via SPDK

Hi In the ceph's source code(https://github.com/ceph/ceph ), we can see: 1. ceph /src /os /bluestore/BlockDevice.cc

Re: [ceph-users] mon osd down out subtree limit default

The docs is providing the following information: The smallest CRUSH unit type that Ceph will notautomatically mark out. For instance, if set to host and if all OSDs of a host are down, Ceph will not automatically mark out these OSDs. But what does it exactly mean? Anyone who can explain it? Than

Re: [ceph-users] migrating cephfs data and metadat to new pools

Why do you need to move the data between pools? My guess is that for your needs you can add another pool to the FS and do something with it that way. Then you are using the same MDS servers and the same FS. I would probably recommend doing the copy using mounted filesystems instead of a rados copy.

[ceph-users] PG reported as inconsistent in status, but no inconsistencies visible to rados

This is an odd one. My cluster is reporting an inconsistent pg in ceph status and ceph health detail. However, rados list-inconsistent-obj and rados list-inconsistent-snapset both report no inconsistencies. Scrubbing the pg results in these errors in the osd logs: OSD 63 (primary): 2017-08-2

[ceph-users] Lots of "wrongly marked me down" messages

Hi, Did you had any further information about this issue? I'm having now the same symptoms with a similar configuration with 240 OSD in a scale-out process. Thanks in advance, Nuno Vargas NOS ___ ceph-users mailing list ceph-users@lists.ceph.com http://

Re: [ceph-users] cluster unavailable for 20 mins when downed server was reintroduced

Hi, On Thu, 17 Aug 2017, Gregory Farnum said: > On Wed, Aug 16, 2017 at 4:04 AM Sean Purdy wrote: > > > On Tue, 15 Aug 2017, Gregory Farnum said: > > > On Tue, Aug 15, 2017 at 4:23 AM Sean Purdy > > wrote: > > > > I have a three node cluster with 6 OSD and 1 mon per node. > > > > > > > > I had

Re: [ceph-users] mon osd down out subtree limit default

Great to hear. Best On Mon, Aug 21, 2017 at 8:54 AM John Spray wrote: > On Mon, Aug 21, 2017 at 4:34 PM, Scottix wrote: > > I don't want to hijack another thread so here is my question. > > I just learned about this option from another thread and from my > > understanding with our Ceph cluster

Re: [ceph-users] mon osd down out subtree limit default

On Mon, Aug 21, 2017 at 4:34 PM, Scottix wrote: > I don't want to hijack another thread so here is my question. > I just learned about this option from another thread and from my > understanding with our Ceph cluster that we have setup, the default value is > not good. Which is "rack" and I should

[ceph-users] mon osd down out subtree limit default

I don't want to hijack another thread so here is my question. I just learned about this option from another thread and from my understanding with our Ceph cluster that we have setup, the default value is not good. Which is "rack" and I should have it on "host". Which comes to my point why is it set

Re: [ceph-users] Any experience with multiple cephfs instances in one ceph cluster? How experimental is this?

On Mon, Aug 21, 2017 at 3:03 PM, Bryan Banister wrote: > Thanks for the response John. > > Maybe I'm not understanding this correctly, but I thought clients could be > restricted to specific file systems by limiting access to the underlying ceph > pools used in each file system? > > client.cephf

Re: [ceph-users] Environment variable to configure rbd "-c" parameter and "--keyfile" parameter?

On Mon, Aug 21, 2017 at 4:11 PM, Ilya Dryomov wrote: > On Mon, Aug 21, 2017 at 2:53 PM, Stéphane Klein > wrote: >> Hi, >> >> I look for environment variable to configure rbd "-c" parameter and >> "--keyfile" parameter. >> >> I found nothing in http://docs.ceph.com/docs/master/man/8/rbd/ > > For -

[ceph-users] migrating cephfs data and metadat to new pools

Assuming that the MDS can be stopped indefinitely, and thus no client access, is there a recommended procedure for migrating the data/metadata pools to new pools? I gather new pools can be created and rados cppool used to migrate the data, but what would be the best way to create the fs with tho

Re: [ceph-users] Environment variable to configure rbd "-c" parameter and "--keyfile" parameter?

On Mon, Aug 21, 2017 at 2:53 PM, Stéphane Klein wrote: > Hi, > > I look for environment variable to configure rbd "-c" parameter and > "--keyfile" parameter. > > I found nothing in http://docs.ceph.com/docs/master/man/8/rbd/ For -c, CEPH_CONF. I don't think there is an environment variable for -

Re: [ceph-users] Any experience with multiple cephfs instances in one ceph cluster? How experimental is this?

Thanks for the response John. Maybe I'm not understanding this correctly, but I thought clients could be restricted to specific file systems by limiting access to the underlying ceph pools used in each file system? client.cephfs.test1 key: AQDuQpdZp90MHhAAkYE6P5XYzsoswgEkZy6RLw==

Re: [ceph-users] Accessing krbd client metrics

On Sat, Aug 19, 2017 at 1:39 AM, Mingliang LIU wrote: > Hi all, > > I have a quick question about the RBD kernel module - how to best collect > the metrics or perf numbers? The command 'ceph -w' does print some useful > event logs of cluster-wide while I'm interested in > per-client/per-image/per-

Re: [ceph-users] ceph pgs state forever stale+active+clean

Thanks for response. I can understand why size of 2 and min_size 1 is not an acceptable in production. but, I just want to make the situation of data loss and to know the health of ceph cluster can be clean in this situation(except data recovery because data is gone). So, I've tried to delete PGs

Re: [ceph-users] Any experience with multiple cephfs instances in one ceph cluster? How experimental is this?

On Mon, Aug 21, 2017 at 2:35 PM, Bryan Banister wrote: > Hi all, > > > > I’m very new to ceph and cephfs, so I’m just starting to play around with > the Luminous release. There are some very concerning warnings about > deploying multiple cephfs instances in the same cluster: > > “There are no kno

[ceph-users] Any experience with multiple cephfs instances in one ceph cluster? How experimental is this?

Hi all, I'm very new to ceph and cephfs, so I'm just starting to play around with the Luminous release. There are some very concerning warnings about deploying multiple cephfs instances in the same cluster: "There are no known bugs, but any failures which do result from having multiple active

Re: [ceph-users] lease_timeout - new election

I really need some help through this. This is happening very frequently and I can't seem to figure out why. My services rely on cephfs and when this happens, the mds suicides. It's always the same, see the last occurrence logs: host bhs1-mail03-ds03: 2017-08-19 06:35:54.072809 7f44c60f1700 1 m

Re: [ceph-users] Exclusive-lock Ceph

Dears,   Some days  ago, I read about this comands rbd lock add  and rbd lock remove , this commands will go maintened in ceph in future versions, or the better form, to use lock in ceph, will go exclusive-lock and this commands will go depreciated ?   Thanks a Lot, Marcelo Em 24/07/2017, Jason

Re: [ceph-users] pros/cons of multiple OSD's per host

It is not recommended to get your cluster more than 70% full due to rebalancing and various other reasons. That would change your 12x 10TB disks in a host to only be 84TB if you filled your cluster to 70% full. I still think that the most important aspects of what is best for you hasn't been provid

[ceph-users] Environment variable to configure rbd "-c" parameter and "--keyfile" parameter?

Hi, I look for environment variable to configure rbd "-c" parameter and "--keyfile" parameter. I found nothing in http://docs.ceph.com/docs/master/man/8/rbd/ Best regards, Stéphane -- Stéphane Klein blog: http://stephane-klein.info cv : http://cv.stephane-klein.info Twitter: http://twitter.com

Re: [ceph-users] Ceph cluster with SSDs

On Mon, 21 Aug 2017 17:13:10 +0530 M Ranga Swami Reddy wrote: > Thank you. > Here I have NVMes from Intel. but as the support of these NVMes not > there from Intel, we decided not to use these NVMes as a journal. You again fail to provide with specific model numbers... No support from Intel sugge

Re: [ceph-users] Ceph cluster with SSDs

Thank you. Here I have NVMes from Intel. but as the support of these NVMes not there from Intel, we decided not to use these NVMes as a journal. Btw, if we split this SSD with multiple OSD (for ex: 1 SSD with 4 or 2 OSDs), is this help any performance numbers? On Sun, Aug 20, 2017 at 9:33 AM, Chr

Re: [ceph-users] ceph pgs state forever stale+active+clean

With the exception of trying to re-add the drive to be able to read the data off of it, your only other option is to accept that you lost data and mark the pg as lost and delete it. Not surprisingly, you can't recover the data without any copies of it. Size of 2 is not an acceptable production seei

Re: [ceph-users] pros/cons of multiple OSD's per host

On Mon, Aug 21, 2017 at 6:40 AM, Nick Tan wrote: > Hi all, > > I'm in the process of building a ceph cluster, primarily to use cephFS. At > this stage I'm in the planning phase and doing a lot of reading on best > practices for building the cluster, however there's one question that I > haven't b

Re: [ceph-users] pros/cons of multiple OSD's per host

On Mon, Aug 21, 2017 at 3:58 PM, Ronny Aasen wrote: > On 21. aug. 2017 07:40, Nick Tan wrote: > >> Hi all, >> >> I'm in the process of building a ceph cluster, primarily to use cephFS. >> At this stage I'm in the planning phase and doing a lot of reading on best >> practices for building the clus

Re: [ceph-users] pros/cons of multiple OSD's per host

On 21. aug. 2017 07:40, Nick Tan wrote: Hi all, I'm in the process of building a ceph cluster, primarily to use cephFS. At this stage I'm in the planning phase and doing a lot of reading on best practices for building the cluster, however there's one question that I haven't been able to find