Re: [ceph-users] librbd on opensolaris/illumos

2016-03-30 Thread Sumit Gaur
Thanks Gregory, For your clear response. Do you see any problem if ceph cluster is used via KVM zone in Opensolaris ? I assume there is no issue to install ceph client on KVM zone in opensolaris. -sumit On Wed, Mar 30, 2016 at 2:47 AM, Gregory Farnum wrote: > On Mon, Mar 28, 2016 at 9:55 PM, Su

Re: [ceph-users] Scrubbing a lot

2016-03-30 Thread Ilya Dryomov
On Wed, Mar 30, 2016 at 3:03 AM, Jason Dillaman wrote: > Understood -- format 2 was promoted to the default image format starting with > Infernalis (which not all users would have played with since it isn't LTS). > The defaults can be overridden via the command-line when creating new images >

Re: [ceph-users] PG Stuck active+undersized+degraded+inconsistent

2016-03-30 Thread Christian Balzer
Hello, On Tue, 29 Mar 2016 18:10:33 + Calvin Morrow wrote: > Ceph cluster with 60 OSDs, Giant 0.87.2. One of the OSDs failed due to a > hardware error, however after normal recovery it seems stuck with > one active+undersized+degraded+inconsistent pg. > Any reason (other than inertia, which

Re: [ceph-users] Redirect snapshot COW to alternative pool

2016-03-30 Thread Nick Fisk
> > > > > I think this is where I see slow performance. If you are doing > > > > large IO, then copying 4MB objects (assuming defaults) is maybe > > > > only 2x times the original IO to the disk. However if you are > > > > doing smaller IO from what I can see a single 4kb write would lead > > > >

[ceph-users] Error mon create-initial

2016-03-30 Thread Mohd Zainal Abidin Rabani
Keep getting this error: [osd04][INFO ] monitor: mon.osd04 is currently at the state of electing [osd04][INFO ] Running command: ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.osd04.asok mon_status [ceph_deploy][ERROR ] GenericError: Failed to create 1 monitors Other osd01, osd02 a

Re: [ceph-users] Radosgw (civetweb) hangs once around 850 established connections

2016-03-30 Thread Dan van der Ster
Hi Sean, Did you check that the process isn't hitting some ulimits? cat /proc/`pidof radosgw`/limits and compare with the num processes/num FDs in use. Cheers, Dan On Tue, Mar 29, 2016 at 8:35 PM, seapasu...@uchicago.edu wrote: > So an update for anyone else having this issue. It looks like ra

Re: [ceph-users] Local SSD cache for ceph on each compute node.

2016-03-30 Thread Nick Fisk
> >> > >> On 03/29/2016 04:35 PM, Nick Fisk wrote: > >>> One thing I picked up on when looking at dm-cache for doing caching > >>> with RBD's is that it wasn't really designed to be used as a > >>> writeback cache for new writes, as in how you would expect a > >>> traditional writeback cache to wor

Re: [ceph-users] Scrubbing a lot

2016-03-30 Thread German Anders
Ok, but I've kernel 3.19.0-39-generic, so the new version is supposed to work right?, and I'm still getting issues while trying to map the RBD: $ *sudo rbd --cluster cephIB create e60host01vX --size 100G --pool rbd -c /etc/ceph/cephIB.conf* $ *sudo rbd -p rbd bench-write e60host01vX --io-size 4096

Re: [ceph-users] Scrubbing a lot

2016-03-30 Thread Jason Dillaman
You will first need to disable all features except for layering since krbd doesn't currently have support: # rbd --cluster cephIB feature disable e60host01vX exclusive-lock,object-map,fast-diff,deep-flatten --pool cinder-volumes You also might want to consider adding "rbd default features = 1"

Re: [ceph-users] Ceph upgrade questions

2016-03-30 Thread Daniel Delin
>Note that 0.94.6 has a massive, data destroying cache-tier bug, so you >will want to wait until .7 at least if you're using cache-tiering, or read >up on the work-around for that bug alternatively. This sounds interesting, is there a bug number for this ? Been playing around with cache tiering i

Re: [ceph-users] Ceph upgrade questions

2016-03-30 Thread Christian Balzer
Hello, On Wed, 30 Mar 2016 16:12:30 +0200 (CEST) Daniel Delin wrote: > > >Note that 0.94.6 has a massive, data destroying cache-tier bug, so you > >will want to wait until .7 at least if you're using cache-tiering, or > >read up on the work-around for that bug alternatively. > > This sounds in

[ceph-users] Incorrect path in /etc/init/ceph-osd.conf?

2016-03-30 Thread Nick Fisk
Hi All, I can see the path in the upstart script https://github.com/ceph/ceph/blob/master/src/upstart/ceph-osd.conf Checks for the file /etc/default/ceph and then runs it But in all the instances of ceph that I have installed that location is a directory and the actual location of t

Re: [ceph-users] PG Stuck active+undersized+degraded+inconsistent

2016-03-30 Thread Calvin Morrow
On Wed, Mar 30, 2016 at 1:27 AM Christian Balzer wrote: > > Hello, > > On Tue, 29 Mar 2016 18:10:33 + Calvin Morrow wrote: > > > Ceph cluster with 60 OSDs, Giant 0.87.2. One of the OSDs failed due to a > > hardware error, however after normal recovery it seems stuck with > > one active+under

Re: [ceph-users] Ceph stopped self repair.

2016-03-30 Thread Gregory Farnum
On Tuesday, March 29, 2016, Dan Moses wrote: > Any suggestions how to clean up ceph errors that don't autocorrect? All > these counters haven't moved in 2 hours now. > > HEALTH_WARN 93 pgs degraded; 93 pgs stuck degraded; 113 pgs stuck unclean; > 93 pgs stuck undersized; 93 pgs undersized; too m

Re: [ceph-users] v10.1.0 Jewel release candidate available

2016-03-30 Thread Alfredo Deza
On Wed, Mar 30, 2016 at 11:49 AM, Xiaoxi Chen wrote: > I am seeing package for precise in > http://download.ceph.com/debian-jewel/dists/precise/, is that > by-accident or we plan to support precise for one more LTS? The configuration is there but there aren't any packages built for it: http://

[ceph-users] ceph pg query hangs for ever

2016-03-30 Thread Wido den Hollander
Hi, I have an issue with a Ceph cluster which I can't resolve. Due to OSD failure a PG is incomplete, but I can't query the PG to see what I can do to fix it. health HEALTH_WARN 1 pgs incomplete 1 pgs stuck inactive 1 pgs stuck unclean 98 requ

Re: [ceph-users] ceph pg query hangs for ever

2016-03-30 Thread Mart van Santen
Hi there, With the help of a lot of people we were able to repair the PG and restored service. We will get back on this later with a full report for future reference. Regards, Mart On 03/30/2016 08:30 PM, Wido den Hollander wrote: > Hi, > > I have an issue with a Ceph cluster which I can't re

[ceph-users] chunk-based cache in ceph with erasure coded back-end storage

2016-03-30 Thread Yu Xiang
Dear List, I am exploring in ceph caching tier recently, considering a cache-tier (replicated) and a back storage-tier (erasure-coded), so chunks are stored in the OSDs in the erasure-coded storage tier, when a file has been requested to read, usually, all chunks in the storage tier would be co

Re: [ceph-users] PG Stuck active+undersized+degraded+inconsistent

2016-03-30 Thread Christian Balzer
On Wed, 30 Mar 2016 15:50:07 + Calvin Morrow wrote: > On Wed, Mar 30, 2016 at 1:27 AM Christian Balzer wrote: > > > > > Hello, > > > > On Tue, 29 Mar 2016 18:10:33 + Calvin Morrow wrote: > > > > > Ceph cluster with 60 OSDs, Giant 0.87.2. One of the OSDs failed due > > > to a hardware er

Re: [ceph-users] chunk-based cache in ceph with erasure coded back-end storage

2016-03-30 Thread huang jun
if your cache-mode is write-back, which will cache the read object in cache tier. you can try the read-proxy mode, which will not cache the object. the read request send to primary OSD, and the primary osd collect the shards from base tier(in you case, is erasure code pool), you need to read at lea

Re: [ceph-users] ceph pg query hangs for ever

2016-03-30 Thread Mart van Santen
Hello, Well unfortunately the problem is not really solved. Yes, we managed to get to a good health state at some point, when a client hits some specific data, the osd process crashes with below errors. The 3 OSD which handle 3.117, the PG with problems, are currently down and reweighted them to

[ceph-users] Ceph.conf

2016-03-30 Thread zainal
Hi, What does mean by mon initial members in ceph.conf? Is it monitor node that monitor all osd node? Or node osd that been monitor? Care to exlain? Regards, Mohd Zainal Abidin Rabani Technical Support ___ ceph-users mailing list ceph-use

[ceph-users] OSD crash after conversion to bluestore

2016-03-30 Thread Adrian Saul
I upgraded my lab cluster to 10.1.0 specifically to test out bluestore and see what latency difference it makes. I was able to one by one zap and recreate my OSDs to bluestore and rebalance the cluster (the change to having new OSDs start with low weight threw me at first, but once I worked t

Re: [ceph-users] Ceph.conf

2016-03-30 Thread Adrian Saul
It is the monitors that ceph clients/daemons can connect to initially to connect with the cluster. Once they connect to one of the initial mons they will get a full list of all monitors and be able to connect to any of them to pull updated maps. From: ceph-users [mailto:ceph-users-boun...@lis

Re: [ceph-users] xenserver or xen ceph

2016-03-30 Thread Jiri Kanicky
Hi. There is a solution for Ceph in XenServer. With the help of my engineer Mark, we developed a simple patch which allows you to search and attach RBD image on XenServer. We create LVHD over the RBD (not RBD per VDI mapping yet), so it is far from ideal, but its a good start. The process of

Re: [ceph-users] Radosgw (civetweb) hangs once around 850 established connections

2016-03-30 Thread seapasu...@uchicago.edu
Thanks Dan! Thanks for this. I didn't know /proc/procid/limits was here! Super useful!! Here are my limits:: root@kh11-9:~# cat /proc/419990/limits Limit Soft Limit Hard Limit Units Max cpu time unlimitedunlimitedsecon

[ceph-users] understand "client rmw"

2016-03-30 Thread Zhongyan Gu
Hi ceph experts, I know rmw means read modify write. Just don't understand what does client rmw stand for. can any body tell me what is it and in what Scenario this kind of requests will be generated. zhongyan ___ ceph-users mailing list ceph-users@lists