[ceph-users] more human readable log to track request or using mapreduce for data statistics

hi，ceph: Currently, the command ”ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops“ may return as below: { "description": "osd_op(client.4436.1:11617 rb.0.1153.6b8b4567.0192 [] 2.8eb4757c ondisk+write e92)", "received_at": "2015-03-25 19:41:47.146145",

Re: [ceph-users] more human readable log to track request or using mapreduce for data statistics

On 26/03/2015, at 09.05, 池信泽 wrote: > hi，ceph: > > Currently, the command ”ceph --admin-daemon > /var/run/ceph/ceph-osd.0.asok dump_historic_ops“ may return as below: > > { "description": "osd_op(client.4436.1:11617 > rb.0.1153.6b8b4567.0192 [] 2.8eb4757c ondisk+write e92)", >

[ceph-users] (no subject)

Thanks for the information. -Sreenath - Date: Wed, 25 Mar 2015 04:11:11 +0100 From: Francois Lafont To: ceph-users Subject: Re: [ceph-users] PG calculator queries Message-ID: <5512274f.1000...@free.fr> Content-Type: text/plain; charset=utf-8 Hi, Sreenath BH wrote : >

[ceph-users] All pools have size=3 but "MB data" and "MB used" ratio is 1 to 5

Thanks for the answer. Now the meaning of "MB data" and "MB used" is clear, and if all the pools have size=3 I expect a ratio 1 to 3 of the two values. I still can't understand why "MB used" is so big in my setup. All my pools are size =3 but the ratio "MB data" and "MB used" is 1 to 5 instead of

[ceph-users] Hammer release data and a Design question

Hi , I 'm just starting on small Ceph implementation and wanted to know the release date for Hammer. Will it coincide with relase of Openstack. My Conf: (using 10G and Jumboframes on Centos 7 / RHEL7 ) 3x Mons (VMs) : CPU - 2 Memory - 4G Storage - 20 GB 4x OSDs : CPU - Haswell Xeon Memory - 8

Re: [ceph-users] Strange osd in PG with new EC-Pool - pgs: 2 active+undersized+degraded

Hi Don, after a lot of trouble due an unfinished setcrushmap, I was able to remove the new EC pool. Load the old crushmap and edit agin. After include an "step set_choose_tries 100" in the crushmap the EC pool creation with ceph osd pool create ec7archiv 1024 1024 erasure 7hostprofile work withou

[ceph-users] How to see the content of an EC Pool after recreate the SSD-Cache tier?

Hi all, due an very silly approach, I removed the cache tier of an filled EC pool. After recreate the pool and connect with the EC pool I don't see any content. How can I see the rbd_data and other files through the new ssd cache tier? I think, that I must recreate the rbd_directory (and fill wit

[ceph-users] running Qemu / Hypervisor AND Ceph on the same nodes

Hi, in the past i rwad pretty often that it's not a good idea to run ceph and qemu / the hypervisors on the same nodes. But why is this a bad idea? You save space and can better use the ressources you have in the nodes anyway. Stefan ___ ceph-users ma

Re: [ceph-users] running Qemu / Hypervisor AND Ceph on the same nodes

On 26-03-15 11:52, Stefan Priebe - Profihost AG wrote: > Hi, > > in the past i rwad pretty often that it's not a good idea to run ceph > and qemu / the hypervisors on the same nodes. > > But why is this a bad idea? You save space and can better use the > ressources you have in the nodes anyway. >

Re: [ceph-users] running Qemu / Hypervisor AND Ceph on the same nodes

Hi Wido, Am 26.03.2015 um 11:59 schrieb Wido den Hollander: > On 26-03-15 11:52, Stefan Priebe - Profihost AG wrote: >> Hi, >> >> in the past i rwad pretty often that it's not a good idea to run ceph >> and qemu / the hypervisors on the same nodes. >> >> But why is this a bad idea? You save space a

Re: [ceph-users] running Qemu / Hypervisor AND Ceph on the same nodes

On 26-03-15 12:04, Stefan Priebe - Profihost AG wrote: > Hi Wido, > Am 26.03.2015 um 11:59 schrieb Wido den Hollander: >> On 26-03-15 11:52, Stefan Priebe - Profihost AG wrote: >>> Hi, >>> >>> in the past i rwad pretty often that it's not a good idea to run ceph >>> and qemu / the hypervisors on th

Re: [ceph-users] more human readable log to track request or using mapreduce for data statistics

On 26/03/2015, at 12.14, 池信泽 wrote: > > It is not so convenience to do conversion in custom. > Because there are many kinds of log in ceph-osd.log. we only need some > of them including latency. > But now, It is hard to grep the log what we want and decode them. Still run output through a pipe wh

Re: [ceph-users] running Qemu / Hypervisor AND Ceph on the same nodes

A word of caution: While normally my OSDs use very little CPU, I have occasionally had an issue where the OSDs saturate the CPU (not necessarily during a rebuild). This might be a kernel thing, or a driver thing specific to our hosts, but were this to happen to you, it now impacts your VMs as well

Re: [ceph-users] running Qemu / Hypervisor AND Ceph on the same nodes

It's kind of a philosophical question. Technically there's nothing that prevents you from putting ceph and the hypervisor on the same boxes. It's a question of whether or not potential cost savings are worth increased risk of failure and contention. You can minimize those things through vario

Re: [ceph-users] How to see the content of an EC Pool after recreate the SSD-Cache tier?

You shouldn't rely on "rados ls" when working with cache pools. It doesn't behave properly and is a silly operation to run against a pool of any size even when it does. :) More specifically, "rados ls" is invoking the "pgls" operation. Normal read/write ops will go query the backing store for obje

[ceph-users] ceph falsely reports clock skew?

I have a virtual test environment of an admin node and 3 mon + osd nodes, built by just following the quick start guide. It seems to work OK but ceph is constantly complaining about clock skew much greater than reality. Clocksource on the virtuals is kvm-clock and they also run ntpd. ceph-admin-n

Re: [ceph-users] All pools have size=3 but "MB data" and "MB used" ratio is 1 to 5

> You just need to go look at one of your OSDs and see what data is > stored on it. Did you configure things so that the journals are using > a file on the same storage disk? If so, *that* is why the "data used" > is large. I followed your suggestion and this is the result of my trobleshooting. E

Re: [ceph-users] All pools have size=3 but "MB data" and "MB used" ratio is 1 to 5

On Thu, Mar 26, 2015 at 2:56 AM, Saverio Proto wrote: > Thanks for the answer. Now the meaning of "MB data" and "MB used" is > clear, and if all the pools have size=3 I expect a ratio 1 to 3 of the > two values. > > I still can't understand why "MB used" is so big in my setup. > All my pools are s

[ceph-users] ceph falsely reports clock skew?

I have a virtual test environment of an admin node and 3 mon + osd nodes, built by just following the quick start guide. It seems to work OK but ceph is constantly complaining about clock skew much greater than reality. Clocksource on the virtuals is kvm-clock and they also run ntpd. ceph-admin-n

Re: [ceph-users] ceph falsely reports clock skew?

On Thu, Mar 26, 2015 at 7:44 AM, Lee Revell wrote: > I have a virtual test environment of an admin node and 3 mon + osd nodes, > built by just following the quick start guide. It seems to work OK but ceph > is constantly complaining about clock skew much greater than reality. > Clocksource on the

Re: [ceph-users] running Qemu / Hypervisor AND Ceph on the same nodes

I run a converged openstack / ceph cluster with 14 1U nodes. Each has 1 SSD (os / journals), 3 1TB spinners (1 OSD each), 16 HT cores, 10Gb NICs for ceph network, and 72GB of RAM. I configure openstack to leave 3GB of RAM unused on each node for OSD / OS overhead. All the VMs are backed by ceph vol

Re: [ceph-users] hadoop namenode not starting due to bindException while deploying hadoop with cephFS

On Wed, Mar 25, 2015 at 8:10 PM, Ridwan Rashid Noel wrote: > Hi Greg, > > Thank you for your response. I have understood that I should be starting > only the mapred daemons when using cephFS instead of HDFS. I have fixed that > and trying to run hadoop wordcount job using this instruction: > > bin

Re: [ceph-users] ceph falsely reports clock skew?

On Thu, 26 Mar 2015, Gregory Farnum wrote: > On Thu, Mar 26, 2015 at 7:44 AM, Lee Revell wrote: > > I have a virtual test environment of an admin node and 3 mon + osd nodes, > > built by just following the quick start guide. It seems to work OK but ceph > > is constantly complaining about clock s

Re: [ceph-users] running Qemu / Hypervisor AND Ceph on the same nodes

I suspect a config like this where you only have 3 OSDs per node would be more manageable than something denser. IE theoretically a single E5-2697v3 is enough to run 36 OSDs in a 4U super micro chassis for a semi-dense converged solution. You could attempt to restrict the OSDs to one socket a

Re: [ceph-users] ceph falsely reports clock skew?

I think I solved the problem. The clock skew only happens when restarting a node to simulate hardware failure. The virtual comes up with a skewed clock and ceph services start before ntp has time to adjust it, then there's a delay before ceph rechecks the clock skew. Lee On Thu, Mar 26, 2015 at 1

Re: [ceph-users] How to see the content of an EC Pool after recreate the SSD-Cache tier?

Hi Greg, ok! It's looks like, that my problem is more setomapval-related... I must o something like rados -p ssd-archiv setomapval rbd_directory name_vm-409-disk-2 "\0x0f\0x00\0x00\0x00"2cfc7ce74b0dc51 but "rados setomapval" don't use the hexvalues - instead of this I got rados -p ssd-archiv li

Re: [ceph-users] Calamari Deployment

For that matter, is there a way to build Calamari without going the whole vagrant path at all? Some way of just building it through command-line tools? I would be building it on an Openstack instance, no GUI. Seems silly to have to install an entire virtualbox environment inside something tha

Re: [ceph-users] Calamari Deployment

> On 26/03/2015, at 17.18, LaBarre, James (CTR) A6IT > wrote: > For that matter, is there a way to build Calamari without going the whole > vagrant path at all? Some way of just building it through command-line > tools? I would be building it on an Openstack instance, no GUI. Seems silly >

Re: [ceph-users] running Qemu / Hypervisor AND Ceph on the same nodes

That one big server sounds great, but it also sounds like a single point of failure. It's also not cheap. I've been able to build this cluster for about $1400 per node, including the 10Gb networking gear, which is less than what I see the _empty case_ you describe going for new. Even used, the lowe

Re: [ceph-users] Calamari Deployment

I used this as a guide for building calamari packages w/o using vagrant. Worked great: http://bryanapperson.com/blog/compiling-calamari-ceph-ubuntu-14-04/ On Thu, Mar 26, 2015 at 10:30 AM, Steffen W Sørensen wrote: > > On 26/03/2015, at 17.18, LaBarre, James (CTR) A6IT < > james.laba...@cigna.co

Re: [ceph-users] running Qemu / Hypervisor AND Ceph on the same nodes

On 03/26/2015 12:13 PM, Quentin Hartman wrote: That one big server sounds great, but it also sounds like a single point of failure. Absolutely, but I'm talking about folks who want dozens of these, not one. It's also not cheap. I've been able to build this cluster for about $1400 per node,

Re: [ceph-users] Calamari Deployment

The first step is incorrect: echo deb http://ppa.launchpad.net/saltstack/salt/ubuntu lsb_release -sc main | sudo tee /etc/apt/sources.list.d/saltstack.list should be echo deb http://ppa.launchpad.net/saltstack/salt/ubuntu $(lsb_release -sc) main | sudo tee /etc/apt/sources.list.d/saltstack.list

Re: [ceph-users] running Qemu / Hypervisor AND Ceph on the same nodes

We run many clusters in a similar config with shared Hypervisor/OSD/RGW/RBD in production and in staging but we have been looking into moving our storage to it's own cluster so that we can scale independently. We used AWS and scaled up a ton of virtual users using JMeter clustering to test performa

Re: [ceph-users] how do I destroy cephfs? (interested in cephfs + tiering + erasure coding)

On 03/25/2015 05:44 PM, Gregory Farnum wrote: On Wed, Mar 25, 2015 at 10:36 AM, Jake Grimmett wrote: Dear All, Please forgive this post if it's naive, I'm trying to familiarise myself with cephfs! I'm using Scientific Linux 6.6. with Ceph 0.87.1 My first steps with cephfs using a replicated

Re: [ceph-users] How to see the content of an EC Pool after recreate the SSD-Cache tier?

I don't know why you're mucking about manually with the rbd directory; the rbd tool and rados handle cache pools correctly as far as I know. -Greg On Thu, Mar 26, 2015 at 8:56 AM, Udo Lembke wrote: > Hi Greg, > ok! > > It's looks like, that my problem is more setomapval-related... > > I must o so

Re: [ceph-users] how do I destroy cephfs? (interested in cephfs + tiering + erasure coding)

For what it's worth, I don't think "being patient" was the answer. I was having the same problem a couple of weeks ago, and I waited from before 5pm one day until after 8am the next, and still got the same errors. I ended up adding a "new" cephfs pool with a newly-created small pool, but was never

Re: [ceph-users] How to see the content of an EC Pool after recreate the SSD-Cache tier?

On 03/26/2015 10:46 AM, Gregory Farnum wrote: I don't know why you're mucking about manually with the rbd directory; the rbd tool and rados handle cache pools correctly as far as I know. That's true, but the rados tool should be able to manipulate binary data more easily. It should probably be

Re: [ceph-users] running Qemu / Hypervisor AND Ceph on the same nodes

Am 26.03.2015 um 16:36 schrieb Mark Nelson: I suspect a config like this where you only have 3 OSDs per node would be more manageable than something denser. IE theoretically a single E5-2697v3 is enough to run 36 OSDs in a 4U super micro chassis for a semi-dense converged solution. You could a

Re: [ceph-users] how do I destroy cephfs? (interested in cephfs + tiering + erasure coding)

There have been bugs here in the recent past which have been fixed for hammer, at least...it's possible we didn't backport it for the giant point release. :( But for users going forward that procedure should be good! -Greg On Thu, Mar 26, 2015 at 11:26 AM, Kyle Hutson wrote: > For what it's wort

[ceph-users] Ceph RBD devices management & OpenSVC integration

Hi Team, I’ve just written blog post regarding integration of CEPH RBD devices management in OpenSVC service : http://www.flox-arts.net/article30/ceph-rbd-devices-management-with-opensvc-service Next blog pos

Re: [ceph-users] Calamari Deployment

Well, we’re a RedHat shop, so I’ll have to see what’s adaptable from there. (Mint on all my home systems, so I’m not totally lost with Ubuntu ) From: Quentin Hartman [mailto:qhart...@direwolfdigital.com] Sent: Thursday, March 26, 2015 1:15 PM To: Steffen W Sørensen Cc: LaBarre, James (CTR) A6IT

Re: [ceph-users] More than 50% osds down, CPUs still busy; will the cluster recover without help?

That's fair enough Greg, I'll keep upgrading when the opportunity arises, and maybe it'll spring back to life someday :-) -Original Message- From: Gregory Farnum [mailto:g...@gregs42.com] Sent: 20 March 2015 23:05 To: Chris Murray Cc: ceph-users Subject: Re: [ceph-users] More than 50% os

[ceph-users] Migrating objects from one pool to another?

Hi, Lately I've been going back to work on one of my first ceph setup and now I see that I have created way too many placement groups for the pools on that setup (about 10 000 too many). I believe this may impact performances negatively, as the performances on this ceph cluster are abysmal. S

Re: [ceph-users] Migrating objects from one pool to another?

> On 26/03/2015, at 20.38, J-P Methot wrote: > > Lately I've been going back to work on one of my first ceph setup and now I > see that I have created way too many placement groups for the pools on that > setup (about 10 000 too many). I believe this may impact performances > negatively, as th

Re: [ceph-users] How to see the content of an EC Pool after recreate the SSD-Cache tier?

Hi Greg, On 26.03.2015 18:46, Gregory Farnum wrote: > I don't know why you're mucking about manually with the rbd directory; > the rbd tool and rados handle cache pools correctly as far as I know. that's because I deleted the cache tier pool, so the files like rbd_header.2cfc7ce74b0dc51 and rbd_d

Re: [ceph-users] Migrating objects from one pool to another?

That's a great idea. I know I can setup cinder (the openstack volume manager) as a multi-backend manager and migrate from one backend to the other, each backend linking to different pools of the same ceph cluster. What bugs me though is that I'm pretty sure the image store, glance, wouldn't let

Re: [ceph-users] Cascading Failure of OSDs

Since I have been in ceph-land today, it reminded me that I needed to close the loop on this. I was finally able to isolate this problem down to a faulty NIC on the ceph cluster network. It "worked", but it was accumulating a huge number of Rx errors. My best guess is some receive buffer cache fail

[ceph-users] All client writes block when 2 of 3 OSDs down

I added the osd pool default min size = 1 to test the behavior when 2 of 3 OSDs are down, but the behavior is exactly the same as without it: when the 2nd OSD is killed, all client writes start to block and these pipe.(stuff).fault messages begin: 2015-03-26 16:08:50.775848 7fce177fe700 0 monclie

Re: [ceph-users] All client writes block when 2 of 3 OSDs down

Has the OSD actually been detected as down yet? You'll also need to set that min size on your existing pools ("ceph osd pool set min_size 1" or similar) to change their behavior; the config option only takes effect for newly-created pools. (Thus the "default".) On Thu, Mar 26, 2015 at 1:29 PM, L

Re: [ceph-users] All client writes block when 2 of 3 OSDs down

On Thu, Mar 26, 2015 at 4:40 PM, Gregory Farnum wrote: > Has the OSD actually been detected as down yet? > > I believe it has, however I can't directly check because "ceph health" starts to hang when I down the second node. > You'll also need to set that min size on your existing pools ("ceph >

Re: [ceph-users] All client writes block when 2 of 3 OSDs down

On Thu, Mar 26, 2015 at 2:30 PM, Lee Revell wrote: > On Thu, Mar 26, 2015 at 4:40 PM, Gregory Farnum wrote: >> >> Has the OSD actually been detected as down yet? >> > > I believe it has, however I can't directly check because "ceph health" > starts to hang when I down the second node. Oh. You ne

Re: [ceph-users] All client writes block when 2 of 3 OSDs down

​Ah, thanks, got it, I wasn't thinking that mons and osds on the same node isn't a likely real world thing. You have to admit that pipe/fault log message is a bit cryptic. Thanks, Lee ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.c

Re: [ceph-users] Migrating objects from one pool to another?

> On 26/03/2015, at 21.07, J-P Methot wrote: > > That's a great idea. I know I can setup cinder (the openstack volume manager) > as a multi-backend manager and migrate from one backend to the other, each > backend linking to different pools of the same ceph cluster. What bugs me > though is t

Re: [ceph-users] Migrating objects from one pool to another?

> On 26/03/2015, at 22.53, Steffen W Sørensen wrote: > >> >> On 26/03/2015, at 21.07, J-P Methot > > wrote: >> >> That's a great idea. I know I can setup cinder (the openstack volume >> manager) as a multi-backend manager and migrate from one backend to the >> oth

Re: [ceph-users] Migrating objects from one pool to another?

On Thu, Mar 26, 2015 at 2:53 PM, Steffen W Sørensen wrote: > >> On 26/03/2015, at 21.07, J-P Methot wrote: >> >> That's a great idea. I know I can setup cinder (the openstack volume >> manager) as a multi-backend manager and migrate from one backend to the >> other, each backend linking to diff

Re: [ceph-users] Migrating objects from one pool to another?

> On 26/03/2015, at 23.01, Gregory Farnum wrote: > > On Thu, Mar 26, 2015 at 2:53 PM, Steffen W Sørensen > wrote: >> >>> On 26/03/2015, at 21.07, J-P Methot wrote: >>> >>> That's a great idea. I know I can setup cinder (the openstack volume >>> manager) as a multi-back

Re: [ceph-users] Migrating objects from one pool to another?

The procedure you've outlined won't copy snapshots, just the head objects. Preserving the proper snapshot metadata and inter-pool relationships on rbd images I think isn't actually possible when trying to change pools. On Thu, Mar 26, 2015 at 3:05 PM, Steffen W Sørensen wrote: > > On 26/03/2015,

Re: [ceph-users] All client writes block when 2 of 3 OSDs down

Greg, Couple of dumb question may be. 1. If you see , the clients are connecting fine with two monitors in the cluster. 2 monitors can never form a quorum, but, 1 can, so, why with 1 monitor (which is I guess happening after making 2 nodes down) it is not able to connect ? 2. Also, my underst

Re: [ceph-users] All client writes block when 2 of 3 OSDs down

On Thu, Mar 26, 2015 at 3:22 PM, Somnath Roy wrote: > Greg, > Couple of dumb question may be. > > 1. If you see , the clients are connecting fine with two monitors in the > cluster. 2 monitors can never form a quorum, but, 1 can, so, why with 1 > monitor (which is I guess happening after making

Re: [ceph-users] All client writes block when 2 of 3 OSDs down

Got most portion of it, thanks ! But, still not able to get when second node is down why with single monitor in the cluster client is not able to connect ? 1 monitor can form a quorum and should be sufficient for a cluster to run. Thanks & Regards Somnath -Original Message- From: Gregor

Re: [ceph-users] All client writes block when 2 of 3 OSDs down

On Thu, Mar 26, 2015 at 3:36 PM, Somnath Roy wrote: > Got most portion of it, thanks ! > But, still not able to get when second node is down why with single monitor > in the cluster client is not able to connect ? > 1 monitor can form a quorum and should be sufficient for a cluster to run. The w

Re: [ceph-users] All client writes block when 2 of 3 OSDs down

Greg, I think you got me wrong. I am not saying each monitor of a group of 3 should be able to change the map. Here is the scenario. 1. Cluster up and running with 3 mons (quorum of 3), all fine. 2. One node (and mon) is down, quorum of 2 , still connecting. 3. 2 nodes (and 2 mons) are down, sh

Re: [ceph-users] All client writes block when 2 of 3 OSDs down

On Thu, Mar 26, 2015 at 3:54 PM, Somnath Roy wrote: > Greg, > I think you got me wrong. I am not saying each monitor of a group of 3 should > be able to change the map. Here is the scenario. > > 1. Cluster up and running with 3 mons (quorum of 3), all fine. > > 2. One node (and mon) is down, quor

Re: [ceph-users] All client writes block when 2 of 3 OSDs down

All clear, thanks :-) -Original Message- From: Gregory Farnum [mailto:g...@gregs42.com] Sent: Thursday, March 26, 2015 3:59 PM To: Somnath Roy Cc: Lee Revell; ceph-users@lists.ceph.com Subject: Re: [ceph-users] All client writes block when 2 of 3 OSDs down On Thu, Mar 26, 2015 at 3:54 PM

Re: [ceph-users] Migrating objects from one pool to another?

> On 26/03/2015, at 23.13, Gregory Farnum wrote: > > The procedure you've outlined won't copy snapshots, just the head > objects. Preserving the proper snapshot metadata and inter-pool > relationships on rbd images I think isn't actually possible when > trying to change pools. This wasn’t ment f

Re: [ceph-users] All client writes block when 2 of 3 OSDs down

> On 26/03/2015, at 23.36, Somnath Roy wrote: > > Got most portion of it, thanks ! > But, still not able to get when second node is down why with single monitor > in the cluster client is not able to connect ? > 1 monitor can form a quorum and should be sufficient for a cluster to run. To have

Re: [ceph-users] Migrating objects from one pool to another?

I thought there was some discussion about this before. Something like creating a new pool and then taking your existing pool as an overlay of the new pool (cache) and then flush the overlay to the new pool. I haven't tried it or know if it is possible. The other option is shut the VM down, create

[ceph-users] Where is the systemd files?

I understand that Giant should have systemd service files, but I don't see them in the CentOS 7 packages. https://github.com/ceph/ceph/tree/giant/systemd [ulhglive-root@mon1 systemd]# rpm -qa | grep --color=always ceph ceph-common-0.93-0.el7.centos.x86_64 python-cephfs-0.93-0.el7.centos.x86_64 li