[ceph-users] Ceph MDS remove

2015-02-24 Thread ceph-users
Hi all, I've set up a ceph cluster using this playbook: https://github.com/ceph/ceph-ansible I've configured in my hosts list [mdss] hostname1 hostname2 I now need to remove this MDS from the cluster. The only document I found is this: http://www.sebastien-han.fr/blog/2012/07/04/remove-a-m

Re: [ceph-users] stuck ceph-deploy mon create-initial / giant

2015-02-24 Thread Stephan Seitz
Hi Loic, this is the content of our ceph.conf [global] fsid = 719f14b2-7475-4b25-8c5f-3ffbcf594d13 mon_initial_members = ceph1, ceph2, ceph3 mon_host = 192.168.10.107,192.168.10.108,192.168.10.109 auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx filestore_x

Re: [ceph-users] Ceph MDS remove

2015-02-24 Thread Xavier Villaneau
Hello, I also had to remove the MDSs on a Giant test cluster a few days ago, and stumbled upon the same problems. Le 24/02/2015 09:58, ceph-users a écrit : Hi all, I've set up a ceph cluster using this playbook: https://github.com/ceph/ceph-ansible I've configured in my hosts list [mdss] ho

Re: [ceph-users] Cluster never reaching clean after osd out

2015-02-24 Thread Stéphane DUGRAVOT
- Mail original - > I have a Cluster of 3 hosts, running Debian wheezy and Backports Kernel > 3.16.0-0.bpo.4-amd64. > For testing I did a > ~# ceph osd out 20 > from a clean state. > Ceph starts rebalancing, watching ceph -w one sees changing pgs stuck unclean > to get up and then go down

Re: [ceph-users] stuck ceph-deploy mon create-initial / giant

2015-02-24 Thread Loic Dachary
On 24/02/2015 09:58, Stephan Seitz wrote: > Hi Loic, > > this is the content of our ceph.conf > > [global] > fsid = 719f14b2-7475-4b25-8c5f-3ffbcf594d13 > mon_initial_members = ceph1, ceph2, ceph3 > mon_host = 192.168.10.107,192.168.10.108,192.168.10.109 > auth_cluster_required = cephx > auth_s

Re: [ceph-users] Ceph MDS remove

2015-02-24 Thread ceph-users
Sorry, forgot to mention that I'm running Ceph 0.87 on Centos 7. On 24/02/2015 10:20, Xavier Villaneau wrote: Hello, I also had to remove the MDSs on a Giant test cluster a few days ago, and stumbled upon the same problems. Le 24/02/2015 09:58, ceph-users a écrit : Hi all, I've set up a ceph

Re: [ceph-users] stuck ceph-deploy mon create-initial / giant

2015-02-24 Thread Christian Balzer
On Tue, 24 Feb 2015 11:17:22 +0100 Loic Dachary wrote: > > > On 24/02/2015 09:58, Stephan Seitz wrote: > > Hi Loic, > > > > this is the content of our ceph.conf > > > > [global] > > fsid = 719f14b2-7475-4b25-8c5f-3ffbcf594d13 > > mon_initial_members = ceph1, ceph2, ceph3 > > mon_host = 192.168

Re: [ceph-users] stuck ceph-deploy mon create-initial / giant

2015-02-24 Thread Loic Dachary
On 24/02/2015 12:00, Christian Balzer wrote: > On Tue, 24 Feb 2015 11:17:22 +0100 Loic Dachary wrote: > >> >> >> On 24/02/2015 09:58, Stephan Seitz wrote: >>> Hi Loic, >>> >>> this is the content of our ceph.conf >>> >>> [global] >>> fsid = 719f14b2-7475-4b25-8c5f-3ffbcf594d13 >>> mon_initial_me

[ceph-users] OSD on LVM volume

2015-02-24 Thread Joerg Henne
Hi all, installing an OSD on an LVM volume seems not to be supported by the current 'ceph-deploy osd' or 'ceph-disk prepare' tools. Therefore I tried to do it manually as suggested here: http://eturnerx.blogspot.de/2014/08/how-i-added-my-lvm-volumes-as-osds-in.html TL;DR, the process is: - creat

[ceph-users] Wrong object and used space count in cache tier pool

2015-02-24 Thread Xavier Villaneau
Hello ceph-users, I am currently making tests on a small cluster, and Cache Tiering is one of those tests. The cluster runs Ceph 0.87 Giant on three Ubuntu 14.04 servers with the 3.16.0 kernel, for a total of 8 OSD and 1 MON. Since there are no SSDs in those servers, I am testing Cache Tierin

Re: [ceph-users] librados - Atomic Write

2015-02-24 Thread Noah Watkins
I'll take a shot at answering this: Operations are atomic in the sense that there are no partial failures. Additionally, access to an object should appear to be serialized. So, two in-flight operations A and B will be applied in either A,B or B,A order. If ordering is important (e.g. the operat

Re: [ceph-users] OSD on LVM volume

2015-02-24 Thread John Spray
On 24/02/2015 12:49, Joerg Henne wrote: This seems to work, however, the disks are not listed by 'ceph-disk list'. Right. ceph-disk uses GPID partition labels to identify the disks. Is there a recommended way of running an OSD on top of a LVM volume? What are the pros and cons of the approac

[ceph-users] Fwd: OSD on LVM volume

2015-02-24 Thread Jörg Henne
2015-02-24 14:05 GMT+01:00 John Spray : > > I imagine that without proper partition labels you'll also not get the > benefit of e.g. the udev magic > that allows plugging OSDs in/out of different hosts. More generally > you'll just be in a rather non standard configuration that will confuse > any

Re: [ceph-users] Ceph MDS remove

2015-02-24 Thread gian
Sorry, forgot to mention that I'm running Ceph 0.87 on Centos 7. On 24/02/2015 10:20, Xavier Villaneau wrote: Hello, I also had to remove the MDSs on a Giant test cluster a few days ago, and stumbled upon the same problems. Le 24/02/2015 09:58, ceph-users a écrit : Hi all, I've set up a ceph

Re: [ceph-users] Ceph MDS remove

2015-02-24 Thread John Spray
On 24/02/2015 09:20, Xavier Villaneau wrote: [root@z-srv-m-cph01 ceph]# ceph mds stat e1: 0/0/0 up 1. question: why the MDS are not stopped? This is just confusing formatting. 0/0/0 means 0 up, 0 in, max_mds=0. This status indicates that you have no filesystem at all. 2. When I try to

Re: [ceph-users] Wrong object and used space count in cache tier pool

2015-02-24 Thread Gregory Farnum
On Tue, Feb 24, 2015 at 6:21 AM, Xavier Villaneau wrote: > Hello ceph-users, > > I am currently making tests on a small cluster, and Cache Tiering is one of > those tests. The cluster runs Ceph 0.87 Giant on three Ubuntu 14.04 servers > with the 3.16.0 kernel, for a total of 8 OSD and 1 MON. > > S

Re: [ceph-users] librados - Atomic Write

2015-02-24 Thread Italo Santos
Hello Noah, In may case the ordering is importante and I seen that librados have an lock implementation which I’ll use that on my implementation. Thanks for your help. Regards. Italo Santos http://italosantos.com.br/ On Tuesday, February 24, 2015 at 12:52, Noah Watkins wrote: > I'll take

Re: [ceph-users] OSD Startup Best Practice: gpt/udev or SysVInit/systemd ?

2015-02-24 Thread Robert LeBlanc
We have had good luck with letting udev do it's thing on CentOS7. On Wed, Feb 18, 2015 at 7:46 PM, Anthony Alba wrote: > Hi Cephers, > > What is your "best practice" for starting up OSDs? > > I am trying to determine the most robust technique on CentOS 7 where I > have too much choice: > > udev/g

Re: [ceph-users] OSD on LVM volume

2015-02-24 Thread Sebastien Han
A while ago, I managed to have this working but this was really tricky. See my comment here: https://github.com/ceph/ceph-ansible/issues/9#issuecomment-37127128 One use case I had was a system with 2 SSD for the OS and a couple of OSDs. Both SSD were in RAID1 and the system was configured with lv

Re: [ceph-users] OSD Performance

2015-02-24 Thread Kevin Walker
Hi All Just recently joined the list and have been reading/learning about ceph for the past few months. Overall it looks to be well suited to our cloud platform but I have stumbled across a few worrying items that hopefully you guys can clarify the status of. Reading through various mailing list

Re: [ceph-users] Ceph MDS remove

2015-02-24 Thread ceph-users
How can I remove the 2nd MDS: # ceph mds dump dumped mdsmap epoch 72 epoch 72 flags 0 created 2015-02-24 15:55:10.631958 modified2015-02-24 17:58:49.400841 tableserver 0 root0 session_timeout 60 session_autoclose 300 max_file_size 1099511627776 last_failure62 last_f

Re: [ceph-users] OSD Performance

2015-02-24 Thread Mark Nelson
On 02/24/2015 04:21 PM, Kevin Walker wrote: Hi All Just recently joined the list and have been reading/learning about ceph for the past few months. Overall it looks to be well suited to our cloud platform but I have stumbled across a few worrying items that hopefully you guys can clarify the sta

Re: [ceph-users] OSD Performance

2015-02-24 Thread Kevin Walker
Hi Mark Thanks for the info, 22k is not bad, but still massively below what a pcie ssd can achieve. Care to expand on why the write IOPS are so low? Was this with a separate RAM disk pcie device or SLC SSD for the journal? That fragmentation percentage looks good. We are considering using just

Re: [ceph-users] OSD Performance

2015-02-24 Thread Mark Nelson
Hi Kevin, Writes are probably limited by a combination of locks, concurrent O_DSYNC journal writes, fsyncs, etc. The tests I mentioned were with both the OSD and the OSD journal on the same PCIe SSD. Others have looked into this in more detail than I have so might be able to chime in with

Re: [ceph-users] More than 50% osds down, CPUs still busy; will the cluster recover without help?

2015-02-24 Thread Gregory Farnum
On Mon, Feb 23, 2015 at 8:59 AM, Chris Murray wrote: > ... Trying to send again after reporting bounce backs to dreamhost ... > ... Trying to send one more time after seeing mails come through the > list today ... > > Hi all, > > First off, I should point out that this is a 'small cluster' issue a

Re: [ceph-users] OSD Performance

2015-02-24 Thread Christian Balzer
On Wed, 25 Feb 2015 02:50:59 +0400 Kevin Walker wrote: > Hi Mark > > Thanks for the info, 22k is not bad, but still massively below what a > pcie ssd can achieve. Care to expand on why the write IOPS are so low? Aside from what Mark mentioned in his reply there's also latency to be considered in

[ceph-users] Centos 7 OSD silently fail to start

2015-02-24 Thread Barclay Jameson
I have tried to install ceph using ceph-deploy but sgdisk seems to have too many issues so I did a manual install. After mkfs.btrfs on the disks and journals and mounted them I then tried to start the osds which failed. The first error was: #/etc/init.d/ceph start osd.0 /etc/init.d/ceph: osd.0 not

Re: [ceph-users] stuck ceph-deploy mon create-initial / giant

2015-02-24 Thread Brad Hubbard
On 02/24/2015 09:06 PM, Loic Dachary wrote: On 24/02/2015 12:00, Christian Balzer wrote: On Tue, 24 Feb 2015 11:17:22 +0100 Loic Dachary wrote: On 24/02/2015 09:58, Stephan Seitz wrote: Hi Loic, this is the content of our ceph.conf [global] fsid = 719f14b2-7475-4b25-8c5f-3ffbcf594d13 mo

Re: [ceph-users] OSD Performance

2015-02-24 Thread Kevin Walker
Hi Christian We are just looking at options at this stage. Using a hardware RAM disk for the journal is the same concept as the SolidFire guys, who are also using XFS (at least they were last time I crossed paths with a customer using SolidFire) and from my experiences with ZFS, using a RAM ba

Re: [ceph-users] re: Upgrade 0.80.5 to 0.80.8 --the VM's read requestbecome too slow

2015-02-24 Thread 杨万元
I compare the applied commits between 0.80.7 and 0.80.8 , then I focus on this: https://github.com/ceph/ceph/commit/711a7e6f81983ff2091caa0f232af914a04a041c?diff=unified#diff-9bcd2f7647a2bd574b6ebe6baf8e61b3 this commits seems take waitfor_read flushed out of the while cycle, maybe this cause th

Re: [ceph-users] OSD Performance

2015-02-24 Thread Christian Balzer
Hello Kevin, On Wed, 25 Feb 2015 07:55:34 +0400 Kevin Walker wrote: > Hi Christian > > We are just looking at options at this stage. > Never a bad thing to do. > Using a hardware RAM disk for the journal is the same concept as the > SolidFire guys, who are also using XFS (at least they were