Re: [ceph-users] Disk failures

Hello, On Thu, 9 Jun 2016 08:43:23 +0200 Gandalf Corvotempesta wrote: > Il 09 giu 2016 02:09, "Christian Balzer" ha scritto: > > Ceph currently doesn't do any (relevant) checksumming at all, so if a > > PRIMARY PG suffers from bit-rot this will be undetected until the next > > deep-scrub. > > >

[ceph-users] RGW integration with keystone

Hi Everyone, I install cluster ceph and while I install it using the keyston options in the ceph.conf cannot get started,but if I remove those options about keystone in the ceph.conf everything is going good. Any ideas? Thanks ___ ceph-users mailing l

Re: [ceph-users] Migrating from one Ceph cluster to another

> Op 8 juni 2016 om 22:49 schreef Marek Dohojda : > > > I have a ceph cluster (Hammer) and I just built a new cluster > (Infernalis). This cluster contains VM boxes based on KVM. > > What I would like to do is move all the data from one ceph cluster to > another. However the only way I could

[ceph-users] not change of journal devices

Hy Cephers. I impletented Ceph with 12 HDDs(for OSD) and 1 SSD(Journal). Device Map is like below (sdb4 is omitted.) /dev/sdc1 is for OSD.0 and /dev/sdb1 is for Journal /dev/sdd1 is for OSD.1 and /dev/sdb2 is for Journal /dev/sde1 is for OSD.2 and /dev/sdb3 is for Journal . . . /dev/sdn1 is for

Re: [ceph-users] Disk failures

2016-06-09 9:16 GMT+02:00 Christian Balzer : > Neither, a journal failure is lethal for the OSD involved and unless you > have LOTS of money RAID1 SSDs are a waste. Ok, so if a journal failure is lethal, ceph automatically remove the affected OSD and start rebalance, right ? > Additionally your c

[ceph-users] RDMA/Infiniband status

Last time i've used Ceph (about 2014) RDMA/Infiniband support was just a proof of concept and I was using IPoIB with low performance (about 8-10GB/s on a Infiniband DDR 20Gb/s) This was 2 years ago. Any news about this? Is RDMA/Infiniband supported like with GlusterFS?

Re: [ceph-users] RDMA/Infiniband status

On Thu, 9 Jun 2016 10:00:33 +0200 Gandalf Corvotempesta wrote: > Last time i've used Ceph (about 2014) RDMA/Infiniband support was just > a proof of concept > and I was using IPoIB with low performance (about 8-10GB/s on a > Infiniband DDR 20Gb/s) > IPoIB is about half the speed of your IB layer,

Re: [ceph-users] RDMA/Infiniband status

2016-06-09 10:18 GMT+02:00 Christian Balzer : > IPoIB is about half the speed of your IB layer, yes. Ok, so it's normal. I've seen benchmarks on net stating that IPoIB on DDR should reach about 16-17Gb/s I'll plan to move to QDR > And bandwidth is (usually) not the biggest issue, latency is. I'v

Re: [ceph-users] Disk failures

Hello, On Thu, 9 Jun 2016 09:59:04 +0200 Gandalf Corvotempesta wrote: > 2016-06-09 9:16 GMT+02:00 Christian Balzer : > > Neither, a journal failure is lethal for the OSD involved and unless > > you have LOTS of money RAID1 SSDs are a waste. > > Ok, so if a journal failure is lethal, ceph automa

Re: [ceph-users] Disk failures

2016-06-09 10:28 GMT+02:00 Christian Balzer : > Define "small" cluster. Max 14 OSD nodes with 12 disks each, replica 3. > Your smallest failure domain both in Ceph (CRUSH rules) and for > calculating how much over-provisioning you need should always be the > node/host. > This is the default CRUSH

[ceph-users] CephFS: mds client failing to respond to cache pressure

Hi, I know this have been asked here a couple of times, but couldn't find anything concrete. I have the following warning in our ceph cluster. mds0: Client web01:cephfs.web01 failing to respond to cache pressure In previous Ceph versions this might have been a bug. But now we are running Jewel.

Re: [ceph-users] un-even data filled on OSDs

Hi Blari, I ran the script and results are below: == ./crush-reweight-by-utilization.py average_util: 0.587024, overload_util: 0.704429, underload_util: 0.587024. reweighted: 43 (0.852690 >= 0.704429) [1.00 -> 0.95] 238 (0.845154 >= 0.704429) [1.00 -> 0.95] 104 (0.827908 >= 0.704429

Re: [ceph-users] OSPF to the host

> Has anybody had any experience with running the network routed down all the > way to the host? > Hello Nick, yes at SWITCH.ch we run OSPF unnumbered on the switches and on the hosts. Each server has two NICs and we are able to plug the servers to any port on the fabric and OSFP will make the m

Re: [ceph-users] CephFS: mds client failing to respond to cache pressure

Hi Elias, When we have received the same warning, our solution has been to increase the inode cache on the MDS. We have added mds cache size = 200 to the [global] section of ceph.conf on the MDS server. We have to restart MDS for the changes to be applied. Sean On 9 June 2016 at 19:55,

Re: [ceph-users] OSPF to the host

> -Original Message- > From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of > Saverio Proto > Sent: 09 June 2016 11:38 > To: n...@fisk.me.uk > Cc: ceph-users@lists.ceph.com > Subject: Re: [ceph-users] OSPF to the host > > > Has anybody had any experience with running th

Re: [ceph-users] CephFS: mds client failing to respond to cache pressure

Hi Sean, Isn't there any downsides to increasing the mds cache size? My colleague mentioned that he tested it previously and then the cluster didn't recover during a failover.. On Thu, Jun 9, 2016 at 12:41 PM, Sean Crosby wrote: > Hi Elias, > > When we have received the same warning, our soluti

Re: [ceph-users] CephFS: mds client failing to respond to cache pressure

Memory usage is all that springs to mind. Our MDS server, with the 2 million inode cache is currently using 2GB RAM. We haven't seen any problems regarding failover (we have 1 active and one failover MDS). Sean On 9 June 2016 at 21:18, Elias Abacioglu wrote: > Hi Sean, > > Isn't there any down

Re: [ceph-users] un-even data filled on OSDs

Swami, Run it with the help option for more context: "./crush-reweight-by-utilization.py --help". In your example below it's reporting to you what changes it would make to your OSD reweight values based on the default option settings (because you didn't specify any options). To make the script act

Re: [ceph-users] Ceph file change monitor

I think you are looking for inotify/fanotify events for Ceph. Usually these are implemented for local file system. Ceph being a networked file system, it will not be easy to implement and will involve network traffic to generate events. Not sure it is in the plan though. Regards, Anand On Wed,

Re: [ceph-users] hadoop on cephfs

You can also have Hadoop talking to the Rados Gateway (SWIFT API) so that the data is in Ceph instead of HDFS. I wrote this tutorial that might help: https://github.com/zioproto/hadoop-swift-tutorial Saverio 2016-04-30 23:55 GMT+02:00 Adam Tygart : > Supposedly cephfs-hadoop worked and/or works

[ceph-users] RGW memory usage

Hello! I have a question regarding Ceph RGW memory usage. We currently have 10 node 1.5 PB raw space cluster with EC profile 6+3. Every node has 29x6TB OSDs and 64 GB of RAM. Recently I've noticed that nodes are starting to suffer from RAM insufficiency. There is currently about 2.6 million files

Re: [ceph-users] RDMA/Infiniband status

IPoIB is done with broadcast packets on the Infiniband fabric. Most switches and opensm (by default) setup a broadcast group at the lowest IB speed (SDR), to support all possible IB connections. If you're using pure DDR, you may need to tune the broadcast group in your subnet manager to set the spe

Re: [ceph-users] RDMA/Infiniband status

Il 09 giu 2016 15:41, "Adam Tygart" ha scritto: > > If you're > using pure DDR, you may need to tune the broadcast group in your > subnet manager to set the speed to DDR. Do you know how to set this with opensm? I would like to bring up my test cluster again next days

Re: [ceph-users] RGW integration with keystone

Hi, what can not get started? have you checked the log files for any critical error message of the “can’t get started component"? what does the config file you have created look like: The good one and the bad one? in your particular case what is your Keystone configuration? And may be provid

Re: [ceph-users] RDMA/Infiniband status

I believe this is what you want: https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Networking_Guide/sec-Configuring_the_Subnet_Manager.html -- Adam On Thu, Jun 9, 2016 at 10:01 AM, Gandalf Corvotempesta wrote: > Il 09 giu 2016 15:41, "Adam Tygart" ha scritto: >> >> I

Re: [ceph-users] not change of journal devices

I made a udev rule for my journal disks. Pull the model from the Mine looks like this: $ cat /etc/udev/rules.d/55-ceph-journals.rules ATTRS{model}=="SDLFNDAR-480G-1H", OWNER="ceph", GROUP="ceph", MODE="660" I got my model by knowing the disk ID the first time and $ udevadm info -n /dev/sdj

[ceph-users] Moving Data from Lustre to Ceph

Dell - Internal Use - Confidential Hello Cephers, Has anybody experimented with moving data from Lustre to Ceph storage? Is there any tools out there you use to do so? Thanks, Hadi ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.cep

Re: [ceph-users] Migrating from one Ceph cluster to another

This is how I did it. I upgraded my old cluster first (live one by one) . Then I added my new OSD servers to my running cluster. Once they were all added I set the weight to 0 on all my original osd's. This causes a lot of IO but all data will be migrated to the new servers. Then you can re

Re: [ceph-users] Migrating from one Ceph cluster to another

>>I am hoping that there is a way to just synch the data (and I do realize that >>KVMs will have to be down for the full migration) from one cluster to >>another. you can also do online , with qemu storage migration blockjob. - Mail original - De: "Marek Dohojda" À: "ceph-users" En

Re: [ceph-users] RDMA/Infiniband status

On 09/06/16 17:01, Gandalf Corvotempesta wrote: > Il 09 giu 2016 15:41, "Adam Tygart" > ha scritto: >> >> If you're >> using pure DDR, you may need to tune the broadcast group in your >> subnet manager to set the speed to DDR. > > Do you know how to set this with opensm? > I

[ceph-users] Issue in creating keyring using cbt.py on a cluster of VMs

Running ./cbt.py commands step by step on a cluster of VMs with Centos 7.2 image on them realized that the following step does not go through: pdsh -R ssh -w root@vm10 ceph-authtool --create-keyring --gen-key --name=mon. /tmp/cbt/ceph/keyring --cap mon 'allow *'ceph-authtool I particularly see th

Re: [ceph-users] Migrating from one Ceph cluster to another

I'd considered a similar migration path in the past (slowly rotate updated osds into the pool and old ones out), but then after watching some of the bugs and discussions regarding ceph cache tiering and the like between giant and hammer/jewel, I was starting to lean more towards the rbd -c oldc

Re: [ceph-users] hadoop on cephfs

On Sat, Apr 30, 2016 at 2:55 PM, Adam Tygart wrote: > Supposedly cephfs-hadoop worked and/or works on hadoop 2. I am in the > process of getting it working with cdh5.7.0 (based on hadoop 2.6.0). > I'm under the impression that it is/was working with 2.4.0 at some > point in time. > > At this very

Re: [ceph-users] RDMA/Infiniband status

Hello, On Thu, 9 Jun 2016 20:28:41 +0200 Daniel Swarbrick wrote: > On 09/06/16 17:01, Gandalf Corvotempesta wrote: > > Il 09 giu 2016 15:41, "Adam Tygart" > > ha scritto: > >> > >> If you're > >> using pure DDR, you may need to tune the broadcast group in your > >> subnet

Re: [ceph-users] un-even data filled on OSDs

Blair - Thanks for the details. I used to set the low priority for recovery during the rebalance/recovery activity. Even though I set the recovery_priority as 5 (instead of 1) and client-op_priority set as 63, some of my customers complained that their VMs are not reachable for a few mins/secs duri

Re: [ceph-users] un-even data filled on OSDs

Hi Swami, That's a known issue, which I believe is much improved in Jewel thanks to a priority queue added somewhere in the OSD op path (I think). If I were you I'd be planning to get off Firefly and upgrade. Cheers, On 10 June 2016 at 12:08, M Ranga Swami Reddy wrote: > Blair - Thanks for the

[ceph-users] Journal partition owner's not change to ceph

Hi Cephers, I hava a ceph cluster Jewel on Ubuntu 16.04. What I am wondering is Whenever I reboot the OSD nodes, the OSD init service is failed. The reason why is that the owner not changed in journal partition. I have btrfs file system and the devices are sdc1,sdd1,sde1.. and so on. They are a

Re: [ceph-users] Journal partition owner's not change to ceph

Hi I think I hit the same issue with the journal partitions when upgrading our cluster to Jewel and Ubuntu 16.04. Our solutions to this issue was the change the GUID partition ID/type of the CEPH journal partition to the official one by hand. Afterwards udev changes the raw journal partitions the