[ceph-users] Failure probability with largish deployments

2013-12-19 Thread Christian Balzer
Hello, In my "Sanity check" thread I postulated yesterday that to get the same redundancy and resilience for disk failures (excluding other factors) as my proposed setup (2 nodes, 2x 11 3TB HDs RAID6 per node, 2 global hotspares, thus 4 OSDs) the "Ceph way" one would need need something like 6 no

Re: [ceph-users] Failure probability with largish deployments

2013-12-19 Thread Wolfgang Hennerbichler
Hello, although I don't know much about this topic, I believe that ceph erasure encoding will probably solve a lot of these issues with some speed tradeoff. With erasure encoding the replicated data eats way less disk capacity, so you could use a higher replication factor with a lower disk usage t

Re: [ceph-users] Failure probability with largish deployments

2013-12-19 Thread Christian Balzer
Hello, On Thu, 19 Dec 2013 09:53:58 +0100 Wolfgang Hennerbichler wrote: > Hello, > > although I don't know much about this topic, I believe that ceph erasure > encoding will probably solve a lot of these issues with some speed > tradeoff. With erasure encoding the replicated data eats way less

Re: [ceph-users] Failure probability with largish deployments

2013-12-19 Thread Robert van Leeuwen
> Yeah, I saw erasure encoding mentioned a little while ago, but that's > likely not to be around by the time I'm going to deploy things. > Nevermind that super bleeding edge isn't my style when it comes to > production systems. ^o^ > And at something like 600 disks, that would still have to be a

Re: [ceph-users] radosgw daemon stalls on download of some files

2013-12-19 Thread Joel van Velden
Yehuda, Do you have any futher detail on this radosgw bug? Does it only apply to emperor? Joel van Velden On 19/12/2013 5:09 a.m., Yehuda Sadeh wrote: We were actually able to find the culprit yesterday. While the nginx workaround might be a valid solution (really depends on who nginx reads fr

Re: [ceph-users] Failure probability with largish deployments

2013-12-19 Thread Mariusz Gronczewski
Dnia 2013-12-19, o godz. 17:39:54 Christian Balzer napisał(a): > > Hello, > > In my "Sanity check" thread I postulated yesterday that to get the > same redundancy and resilience for disk failures (excluding other > factors) as my proposed setup (2 nodes, 2x 11 3TB HDs RAID6 per node, > 2 global

Re: [ceph-users] Failure probability with largish deployments

2013-12-19 Thread Wido den Hollander
On 12/19/2013 09:39 AM, Christian Balzer wrote: Hello, In my "Sanity check" thread I postulated yesterday that to get the same redundancy and resilience for disk failures (excluding other factors) as my proposed setup (2 nodes, 2x 11 3TB HDs RAID6 per node, 2 global hotspares, thus 4 OSDs) the

Re: [ceph-users] radosgw setup - not working

2013-12-19 Thread raj kumar
Working now. Removed escape char from the api_key, Thank you so much for your suggestions. On Thu, Dec 19, 2013 at 4:31 AM, Andrew Woodward wrote: > I see python 2.6 So I assume this is a RHEL6 distro. I've not been able to > use mod-fcgid in any setup on RHEL6 variants. I'd reccomend you use

[ceph-users] Armel debian repository

2013-12-19 Thread Mario Giammarco
Hello, I would like to install ceph on a Netgear ReadyNAS 102. It is a debian wheezy based. I have tried to add ceph repository but nas is "armel" architecture and I see you provide a repo for "armhf" architecture. How can I solve this problem? Thanks, Mario _

[ceph-users] [Rados Bench Result] Maximum bandwidth only 380MB/sec with 10Gb Network.

2013-12-19 Thread Kuo Hugo
Hi folks, I'm doing a test for Rados bench now. The cluster is deployed by ceph-deploy Ceph version: Emperor FS : XFS I created a pool test3 with size1 : pool 13 'test3' rep size 1 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 2000 pgp_num 2000 last_change 166 owner 0 The Rados bench

[ceph-users] Storing VM Images on CEPH with RBD-QEMU driver

2013-12-19 Thread Peder Jansson
Hi, I'm testing CEPH with the RBD/QEMU driver through libvirt to store my VM images on. Installation and configuration all went very well with the ceph-deploy tool. I have set up authx authentication in libvirt and that works like a charm too. However, when coming to performance I have big iss

Re: [ceph-users] ceph-deploy: cluster network and admin node

2013-12-19 Thread Gandalf Corvotempesta
2013/12/17 Gandalf Corvotempesta : > There isnt' anything about how to define a cluster network for OSD. > I don't know how to set a cluster address to each OSD. No help about this? I would like to set a cluster-address for each OSD Is this possible with ceph-deploy ? _

Re: [ceph-users] Failure probability with largish deployments

2013-12-19 Thread Gregory Farnum
On Thu, Dec 19, 2013 at 12:39 AM, Christian Balzer wrote: > > Hello, > > In my "Sanity check" thread I postulated yesterday that to get the same > redundancy and resilience for disk failures (excluding other factors) as > my proposed setup (2 nodes, 2x 11 3TB HDs RAID6 per node, 2 > global hotspar

Re: [ceph-users] [Rados Bench Result] Maximum bandwidth only 380MB/sec with 10Gb Network.

2013-12-19 Thread Shain Miley
Three things come to my mind when looking at your setup/results: 1)The number of pg's. According to the documentation it should be: (number of OSD * 100 ) / number of replicas. Maybe playing with the number a bit would yield better results. 2) Although I am NOT using SSD's as journals, I am u

[ceph-users] Add a noter node with odd ans monitor

2013-12-19 Thread Julien Calvet
Hello, When I try to deploy a new monitor on a new node with ceph-deploy, I have this error: — eph@p1:~$ ceph-deploy mon create s4.13h.com [ceph_deploy.cli][INFO ] Invoked (1.3.3): /usr/bin/ceph-deploy mon create s4.13h.com [ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts s4.13h.c

Re: [ceph-users] Failure probability with largish deployments

2013-12-19 Thread Gruher, Joseph R
>-Original Message- >From: ceph-users-boun...@lists.ceph.com [mailto:ceph-users- >boun...@lists.ceph.com] On Behalf Of Gregory Farnum >Sent: Thursday, December 19, 2013 7:20 AM >To: Christian Balzer >Cc: ceph-users@lists.ceph.com >Subject: Re: [ceph-users] Failure probability with largish d

Re: [ceph-users] Error connecting to ceph cluster in openstack cinder

2013-12-19 Thread bigbird Lim
Hi Don, ceph.conf is readable by all users Thanks Song On Wed, Dec 18, 2013 at 10:19 AM, Don Talton (dotalton) wrote: > Check that cinder has access to read your ceph.conf file. I’ve had to > 644 mine. > > > > *From:* ceph-users-boun...@lists.ceph.com [mailto: > ceph-users-boun...@lists.ceph.

[ceph-users] When will Ceph FS be ready for use with production data

2013-12-19 Thread Abhijeet Nakhwa
Ceph FS is really cool and exciting! It makes a lot of sense for us to leverage it. Is there any established goal / timelines for using Ceph FS for production use? Are specific individual support contracts available if Ceph FS is to be used in production? Thanks!

Re: [ceph-users] Failure probability with largish deployments

2013-12-19 Thread Wolfgang Hennerbichler
On 19 Dec 2013, at 16:43, Gruher, Joseph R wrote: > It seems like this calculation ignores that in a large Ceph cluster with > triple replication having three drive failures doesn't automatically > guarantee data loss (unlike a RAID6 array)? not true with RBD images, which are potentially stri

Re: [ceph-users] Failure probability with largish deployments

2013-12-19 Thread Wido den Hollander
On 12/19/2013 08:39 PM, Wolfgang Hennerbichler wrote: On 19 Dec 2013, at 16:43, Gruher, Joseph R wrote: It seems like this calculation ignores that in a large Ceph cluster with triple replication having three drive failures doesn't automatically guarantee data loss (unlike a RAID6 array)?

Re: [ceph-users] When will Ceph FS be ready for use with production data

2013-12-19 Thread Wido den Hollander
On 12/19/2013 08:07 PM, Abhijeet Nakhwa wrote: Ceph FS is really cool and exciting! It makes a lot of sense for us to leverage it. Is there any established goal / timelines for using Ceph FS for production use? Are specific individual support contracts available if Ceph FS is to be used in prod

Re: [ceph-users] Failure probability with largish deployments

2013-12-19 Thread Johannes Formann
Am 19.12.2013 um 20:39 schrieb Wolfgang Hennerbichler : > On 19 Dec 2013, at 16:43, Gruher, Joseph R wrote: > >> It seems like this calculation ignores that in a large Ceph cluster with >> triple replication having three drive failures doesn't automatically >> guarantee data loss (unlike a RAID

[ceph-users] ceph-deploy issues with initial mons that aren't up

2013-12-19 Thread Don Talton (dotalton)
Hi all, I've been working in some ceph-deploy automation and think I've stumbled on an interesting behavior. I create a new cluster, and specify 3 machines. If all 3 are not and unable to be ssh'd into with the account I created for ceph-deploy, then the mon create process will fail and the clu

[ceph-users] admin user for the object gateway?

2013-12-19 Thread Blair Nilsson
How do find or create a user that can use the admin operations for the object gateway? The manual says "Some operations require that the user holds special administrative capabilities." But I can't find if there is a pre setup user with these, or how to create one myself.

Re: [ceph-users] Performance questions (how original, I know)

2013-12-19 Thread Mark Nelson
On 12/16/2013 02:42 AM, Christian Balzer wrote: Hello, Hi Christian! new to Ceph, not new to replicated storage. Simple test cluster with 2 identical nodes running Debian Jessie, thus ceph 0.48. And yes, I very much prefer a distro supported package. I know you'd like to use the distro pa

[ceph-users] rebooting nodes in a ceph cluster

2013-12-19 Thread John-Paul Robinson
What impact does rebooting nodes in a ceph cluster have on the health of the ceph cluster? Can it trigger rebalancing activities that then have to be undone once the node comes back up? I have a 4 node ceph cluster each node has 11 osds. There is a single pool with redundant storage. If it take

Re: [ceph-users] rebooting nodes in a ceph cluster

2013-12-19 Thread Sage Weil
On Thu, 19 Dec 2013, John-Paul Robinson wrote: > What impact does rebooting nodes in a ceph cluster have on the health of > the ceph cluster? Can it trigger rebalancing activities that then have > to be undone once the node comes back up? > > I have a 4 node ceph cluster each node has 11 osds. T

Re: [ceph-users] rebooting nodes in a ceph cluster

2013-12-19 Thread David Clarke
On 20/12/13 13:51, Sage Weil wrote: > On Thu, 19 Dec 2013, John-Paul Robinson wrote: >> What impact does rebooting nodes in a ceph cluster have on the health of >> the ceph cluster? Can it trigger rebalancing activities that then have >> to be undone once the node comes back up? >> >> I have a 4 n

Re: [ceph-users] rebooting nodes in a ceph cluster

2013-12-19 Thread John-Paul Robinson (Campus)
So is it recommended to adjust the rebalance timeout to align with the time to reboot individual nodes? I didn't see this in my pass through the ops manual but maybe I'm not looking in the right place. Thanks, ~jpr > On Dec 19, 2013, at 6:51 PM, "Sage Weil" wrote: > >> On Thu, 19 Dec 20

Re: [ceph-users] Performance questions (how original, I know)

2013-12-19 Thread Christian Balzer
Hello Mark, On Thu, 19 Dec 2013 17:18:01 -0600 Mark Nelson wrote: > On 12/16/2013 02:42 AM, Christian Balzer wrote: > > > > Hello, > > Hi Christian! > > > > > new to Ceph, not new to replicated storage. > > Simple test cluster with 2 identical nodes running Debian Jessie, thus > > ceph 0.48. A

Re: [ceph-users] radosgw daemon stalls on download of some files

2013-12-19 Thread Kyle Bader
> Do you have any futher detail on this radosgw bug? https://github.com/ceph/ceph/commit/0f36eddbe7e745665a634a16bf3bf35a3d0ac424 https://github.com/ceph/ceph/commit/0b9dc0e5890237368ba3dc34cb029010cb0b67fd > Does it only apply to emperor? The bug is present in dumpling too.

Re: [ceph-users] Failure probability with largish deployments

2013-12-19 Thread Christian Balzer
Hello, On Thu, 19 Dec 2013 12:12:13 +0100 Mariusz Gronczewski wrote: > Dnia 2013-12-19, o godz. 17:39:54 > Christian Balzer napisał(a): [snip] > > > > So am I completely off my wagon here? > > How do people deal with this when potentially deploying hundreds of > > disks in a single cluster/p

Re: [ceph-users] Failure probability with largish deployments

2013-12-19 Thread Christian Balzer
Hello, On Thu, 19 Dec 2013 12:42:15 +0100 Wido den Hollander wrote: > On 12/19/2013 09:39 AM, Christian Balzer wrote: [snip] > > > > I'd suggest to use different vendors for the disks, so that means you'll > probably be mixing Seagate and Western Digital in such a setup. > That's funny, becaus

Re: [ceph-users] Failure probability with largish deployments

2013-12-19 Thread Christian Balzer
Hello, On Thu, 19 Dec 2013 15:43:16 + Gruher, Joseph R wrote: [snip] > > It seems like this calculation ignores that in a large Ceph cluster with > triple replication having three drive failures doesn't automatically > guarantee data loss (unlike a RAID6 array)? If your data is triple > re

Re: [ceph-users] Failure probability with largish deployments

2013-12-19 Thread Christian Balzer
On Thu, 19 Dec 2013 21:01:47 +0100 Wido den Hollander wrote: > On 12/19/2013 08:39 PM, Wolfgang Hennerbichler wrote: > > On 19 Dec 2013, at 16:43, Gruher, Joseph R > > wrote: > > > >> It seems like this calculation ignores that in a large Ceph cluster > >> with triple replication having three dri

Re: [ceph-users] ceph-deploy issues with initial mons that aren't up

2013-12-19 Thread Don Talton (dotalton)
I just realized my email is not clear. If the first mon is up and the additional initials are not, then the process fails. > -Original Message- > From: ceph-users-boun...@lists.ceph.com [mailto:ceph-users- > boun...@lists.ceph.com] On Behalf Of Don Talton (dotalton) > Sent: Thursday, Dece

Re: [ceph-users] ceph-deploy issues with initial mons that aren't up

2013-12-19 Thread Gregory Farnum
"mon initial members" is a race prevention mechanism whose purpose is to prevent your monitors from forming separate quorums when they're brought up by automated software provisioning systems (by not allowing monitors to form a quorum unless everybody in the list is a member). If you want to add ot

Re: [ceph-users] admin user for the object gateway?

2013-12-19 Thread JuanJose Galvez
On 12/19/2013 2:02 PM, Blair Nilsson wrote: > How do find or create a user that can use the admin operations for the > object gateway? > > The manual says "Some operations require that the user holds special > administrative capabilities." > > But I can't find if there is a pre setup user with the

Re: [ceph-users] Storing VM Images on CEPH with RBD-QEMU driver

2013-12-19 Thread Wido den Hollander
On 12/19/2013 04:00 PM, Peder Jansson wrote: Hi, I'm testing CEPH with the RBD/QEMU driver through libvirt to store my VM images on. Installation and configuration all went very well with the ceph-deploy tool. I have set up authx authentication in libvirt and that works like a charm too. Howeve

Re: [ceph-users] Ceph network topology with redundant switches

2013-12-19 Thread Wido den Hollander
On 12/18/2013 09:39 PM, Tim Bishop wrote: Hi all, I'm investigating and planning a new Ceph cluster starting with 6 nodes with currently planned growth to 12 nodes over a few years. Each node will probably contain 4 OSDs, maybe 6. The area I'm currently investigating is how to configure the net