Re: [ceph-users] Help

Hi Ashraful, You can try this to have a test ceph cluster with one monitor and two osds running on single machine. I personally use a 2GB virtual machine running ubuntu 12.04, but it works fine even if you have 1GB VM. https://gist.github.com/sorenh/9366800 Its a bit modified version of Loic'

Re: [ceph-users] Python exception on ceph-deploy admin

Hi , Have you created /etc/ceph directory in ur "domUca" node. Seems to me this directory is missing. And while installing ceph from quick start we are not supposed to install ceph on admin node. If the above directory is missing , please create one and run the command.. ceph-deploy admin domUca.

[ceph-users] Does anyone succeed installing ceph RGW in the CentOS?

OS: CentOS 6.4 version: Ceph 0.67.7 I have installed ceph in 3 servers. I can mount to ceph, upload and download files. But when I tried to configure the RGW, I found the document was written for ubuntu. There may be some mistakes, such as "/etc/init.d/radosgw start". There is not /etc/init.d/

[ceph-users] rbd format 2 && stripe-count != 1 cannot be mapped with rbd.ko kernel 3.13.5

OK, it seems during the rbd creation with --stripe-count != 1 you have to follow the rule: stripe-unit * stripe-count = object-size So, EINVAL below was correct, but the message "librbd: STRIPINGV2 and format 2 or later required for non-default striping" is wrong and confusing. Now I have the p

Re: [ceph-users] Does anyone succeed installing ceph RGW in the CentOS?

Hi! I had also a lot of trouble making it work! First of all edit the first lines of /etc/init.d/ceph-radosgw script to make it look like the following #! /bin/bash set -x What "set -x" does is that it puts it in verbose mode and you get more info when you try to start the service. After y

Re: [ceph-users] Python exception on ceph-deploy admin

Thanks for the pointer about not installing Ceph on the admin node. Should I explicitly remove it? Regarding /etc/ceph, it's there: ceph@domUca s2cCluster]$ ls -la /etc/ceph total 20 drwxr-xr-x. 2 root root 4096 Mar 12 01:02 . drwxr-xr-x. 77 root root 4096 Mar 12 03:41 .. -rw-r--r--. 1 root r

Re: [ceph-users] rbd format 2 && stripe-count != 1 cannot be mapped with rbd.ko kernel 3.13.5

Try with --pool instead of -p Sent from my mobile device. Please excuse brevity and typographical errors. On Mar 12, 2014 5:51 AM, "Kasper Dieter" wrote: > OK, > it seems during the rbd creation with --stripe-count != 1 > you have to follow the rule: stripe-unit * stripe-count = object-size > >

Re: [ceph-users] rbd format 2 && stripe-count != 1 cannot be mapped with rbd.ko kernel 3.13.5

Disregard... I just saw your dmesg output.. Sent from my mobile device. Please excuse brevity and typographical errors. On Mar 12, 2014 7:51 AM, "Michael J. Kidd" wrote: > Try with --pool instead of -p > > Sent from my mobile device. Please excuse brevity and typographical > errors. > On Mar 1

Re: [ceph-users] rbd format 2 && stripe-count != 1 cannot be mapped with rbd.ko kernel 3.13.5

No difference: [root@rx37-0 ~]# rbd map --pool SSD-r2 t2-1 rbd: add failed: (22) Invalid argument [root@rx37-0 ~]# dmesg | tail (…) rbd: image t2-1: unsupported stripe unit (got 1048576 want 4194304) The strace of ‘rbd map’ says: open("/sys/bus/rbd/add", O_WRONLY) = 3 write(3, "192.168.113.13:67

Re: [ceph-users] Python exception on ceph-deploy admin

On Tue, Mar 11, 2014 at 9:12 PM, Mark s2c wrote: > > Hello > I followed everything in the set up documentation setting up a test cluster > on an XCP install and got this: > > Invoked (1.3.5): /usr/bin/ceph-- > deploy admin domUs1 domUs2 domUs3 domUca > Pushing admin keys and conf to domUs1 > conn

Re: [ceph-users] rbd format 2 && stripe-count != 1 cannot be mapped with rbd.ko kernel 3.13.5

On Wed, Mar 12, 2014 at 2:02 PM, Kasper, Dieter wrote: > No difference: > > > > [root@rx37-0 ~]# rbd map --pool SSD-r2 t2-1 > > rbd: add failed: (22) Invalid argument > > > > [root@rx37-0 ~]# dmesg | tail > > (...) > > rbd: image t2-1: unsupported stripe unit (got 1048576 want 4194304) > > > > The

Re: [ceph-users] rbd-fuse rbd_list: error %d Numerical result out of range

On Mon, Mar 10, 2014 at 12:32 PM, Ilya Dryomov wrote: > On Mon, Mar 10, 2014 at 12:25 PM, Ilya Dryomov > wrote: >> Hi Graeme, >> >> It looks like not enough memory is allocated for image names. On top >> of that, error reporting could have been better. What's the output of >> 'rbd ls | wc -lc'

Re: [ceph-users] how to configure ceph object gateway

Hi, what commands are “not found”? This page for configuring the RGW works fine as far as I know as I used it no later than a week ago. Can you please give us more details? What is your layout (radosgw installed on a ceph node, mon node, standalone node)? Note: In order to get it running, rem

[ceph-users] How to use ceph

Hi, My name is Ashraful Arefeen. I want to use ceph for testing purpose. Is it possible to use it in a single machine (I mean in one computer)? If it is possible then what will be the preferable configuration of the computer and in that case what are required software apart from ceph? I have start

Re: [ceph-users] rbd create ... STRIPINGV2 and format 2 or later required

Hi Dieter, you have a problem with your command. You set order = 16 so your RBD objects is going to be 65536 bytes Then you tell RBD that you stripe-unit is going to be 65536 which is the size of your full object. Either decrease the size of —stripe-unit to 8192 for example Or increase order s

[ceph-users] Some Questions about using ceph with VMware

Hello everyone, this is the first time i ever write to a mailing list, please be patient with me (especially for my poor english)… Im trying to reach my Bachelors Degree in Computer Science, Im doing a Project which involves ceph. I’am able to setup a ceph Cluster, but there are a few things i c

Re: [ceph-users] rbd format 2 && stripe-count != 1 cannot be mapped with rbd.ko kernel 3.13.5

Please see this Email on ceph-devel ---snip--- Date: Thu, 15 Aug 2013 14:30:24 +0200 From: Damien Churchill To: "Kasper, Dieter" CC: "ceph-de...@vger.kernel.org" Subject: Re: rbd: format 2 support in rbd.ko ? On 15 August 2013 12:42, Kasper Dieter wrote: > When will 'format 2' support be avail

Re: [ceph-users] Some Questions about using ceph with VMware

On Wednesday, March 12, 2014, Florian Krauß wrote: > Hello everyone, > > this is the first time i ever write to a mailing list, please be patient > with me (especially for my poor english)... > Im trying to reach my Bachelors Degree in Computer Science, Im doing a > Project which involves ceph. >

[ceph-users] Write performance from qemu rbd (Mysql)

Hi, Are there any people running Mysql instances from qemu rbd? The performance of the databases is not that great. I've been running some tests in the past and the performance of Mysql seem to be on par with my benchmarks when I'm running benchmarks single threaded. Does anyone have experience

Re: [ceph-users] Write performance from qemu rbd (Mysql)

Hi Robert, > Are there any people running Mysql instances from qemu rbd? > The performance of the databases is not that great. try to setup bcache or flashcache on top of the rdb-device. With a Intel S37000 caching SSD-disk, we got the following fio results agains a bcache'd rbd device:

Re: [ceph-users] PG Scaling

Correction: Sorry min_size is at 1 everywhere. Thank you. Karol Kozubal From: Karol Kozubal mailto:karol.kozu...@elits.com>> Date: Wednesday, March 12, 2014 at 12:06 PM To: "ceph-users@lists.ceph.com" mailto:ceph-users@lists.ceph.com>> Subject: PG Scaling Hi

[ceph-users] PG Scaling

Hi Everyone, I am deploying an openstack deployment with Fuel 4.1 and have a 20 node ceph deployment of c6220’s with 3 osd’s and 1 journaling disk per node. When first deployed each storage pool is configured with the correct size and min_size attributes however fuel doesn’t seem to apply the c

[ceph-users] First Ceph Athens Meetup!

Hello everybody, we are happy to invite you to the first Ceph Athens meetup: http://www.meetup.com/Ceph-Athens on March 18th, 19:30, taking place on the 4th floor of the GRNET [1] HQ offices. We'll be happy to have Steve Starbuck of Inktank with us, who will introduce Ceph. Also, Vangelis Kouki

Re: [ceph-users] clock skew

2014-01-30 18:41 GMT+01:00 Eric Eastman : > I have this problem on some of my Ceph clusters, and I think it is due to > the older hardware the I am using does not have the best clocks. To fix the > problem, I setup one server in my lab to be my local NTP time server, and > then on each of my Ceph

[ceph-users] Wrong PG nums

Hi to all I have this in my conf: # grep 'pg num' /etc/ceph/ceph.conf osd pool default pg num = 5600 But: # ceph osd pool get data pg_num pg_num: 64 Is this normal ? Why just 64 pg was created ? ___ ceph-users mailing list ceph-users@lists.ceph.com

Re: [ceph-users] First Ceph Athens Meetup!

Hi Constantinos, I've added it to https://wiki.ceph.com/Community/Meetups . Feel free to update it if I made a mistake ;-) Cheers On 12/03/2014 17:40, Constantinos Venetsanopoulos wrote: > Hello everybody, > > we are happy to invite you to the first Ceph Athens meetup: > > http://www.meetup.c

[ceph-users] neither POSIX nor REST

Hi, I saw radosgw-admin doesn't require radosgw daemon. I was wondering if by adding creation of buckets and manipulation of objects to radosgw-admin (or a library similar to it) ceph would have a great tool between the low and high levels of rados and radosgw, and simpler than cephfs. Of course

Re: [ceph-users] clock skew

On Mar 12, 2014, at 10:44 AM, Gandalf Corvotempesta wrote: > 2014-01-30 18:41 GMT+01:00 Eric Eastman : >> I have this problem on some of my Ceph clusters, and I think it is due to >> the older hardware the I am using does not have the best clocks. To fix the >> problem, I setup one server in my

[ceph-users] radosgw where to put

Hello to everybody, does anbody know, if I need separate nodes to install radosgw (for example, to place into DMZ), or should I put radosgw daemons directly onto my storage nodes? Thank you in advance! Best wisches, Sergejs___ ceph-users mailing list c

[ceph-users] Put Ceph Cluster Behind a Pair of LB

Hi everyone, Anybody has a good practice on how to set up a ceph cluster behind a pair of load balancer? signature.asc Description: Message signed with OpenPGP using GPGMail ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/l

Re: [ceph-users] Put Ceph Cluster Behind a Pair of LB

>From what I understand about Ceph architecture you would be causing a bottleneck for your ceph traffic. Ceph advantage is the potential concurrency of the traffic and the decentralization of the client facing interfaces increasing scale-out capabilities. Can you give a bit more details about your

Re: [ceph-users] neither POSIX nor REST

On Wed, Mar 12, 2014 at 9:58 AM, Lorieri wrote: > Hi, > > I saw radosgw-admin doesn't require radosgw daemon. > > I was wondering if by adding creation of buckets and manipulation of > objects to radosgw-admin (or a library similar to it) ceph would have > a great tool between the low and high lev

[ceph-users] "no user info saved" after user creation / can't create buckets

Rados GW and Ceph versions installed: Version: 0.67.7-1precise I create a user: radosgw-admin --name client.radosgw. user create --uid test --display-name "Test User" It outputs some JSON that looks convincing: { "user_id": "test", "display_name": "test user", "email": "", "suspended": 0,

Re: [ceph-users] PG Scaling

Round up your pg_num and pgp_num to the next power of 2, 2048. Ceph will start moving data as soon as you implement the new 'size 3', so I would increase the pg_num and pgp_num, first, then increase the size. It will start creating the new PG's immediately. You can see all this going on using

Re: [ceph-users] "no user info saved" after user creation / can't create buckets

Also... what are linger_ops? ceph --admin-daemon /var/run/ceph/ceph-client.radosgw..asok objecter_requests { "ops": [], "linger_ops": [ { "linger_id": 1, "pg": "7.4322fa9f", "osd": 25, "object_id": "notify.0", "object_locator": "@7", "sna

Re: [ceph-users] Put Ceph Cluster Behind a Pair of LB

> Anybody has a good practice on how to set up a ceph cluster behind a pair of > load balancer? The only place you would want to put a load balancer in the context of a Ceph cluster would be north of RGW nodes. You can do L3 transparent load balancing or balance with a L7 proxy, ie Linux Virtual

Re: [ceph-users] Put Ceph Cluster Behind a Pair of LB

You're right. Sorry didn't specify I was trying this for Radosgw. Even for this I'm seeing performance degrade once my clients start to hit the LB VIP. On Mar 12, 2014, at 10:57 AM, Kyle Bader wrote: >> Anybody has a good practice on how to set up a ceph cluster behind a pair of >> load bala

Re: [ceph-users] Put Ceph Cluster Behind a Pair of LB

> You're right. Sorry didn't specify I was trying this for Radosgw. Even for > this I'm seeing performance degrade once my clients start to hit the LB VIP. Could you tell us more about your load balancer and configuration? -- Kyle ___ ceph-users ma

Re: [ceph-users] Put Ceph Cluster Behind a Pair of LB

This is in my lab. Plain passthrough setup with automap enabled on the F5. s3 & curl work fine as far as queries go. But file transfer rate degrades badly once I start file up/download. On Mar 12, 2014, at 11:12 AM, Kyle Bader wrote: >> You're right. Sorry didn't specify I was trying this for

[ceph-users] if partition name changes, will ceph get corrupted?

If a partition name such as "/dev/sdd" changes to "/dev/sde" and ceph was already mapped to the old "/dev/sdd", how will CEPH react? For example, would it get corrupted, or notice a problem and fail remove that one OSD from cluster or somehow automatically re-adapt? FYI: I ask because we added a n

Re: [ceph-users] if partition name changes, will ceph get corrupted?

We use /dev/disk/by-path for this reason, but we confirmed that is stable for our HBAs. Maybe /dev/disk/by-something is consistent with your controller. Cheers, Dan -- Dan van der Ster || Data & Storage Services || CERN IT Department -- Original Message From: Sidharta Mukerjee

Re: [ceph-users] Put Ceph Cluster Behind a Pair of LB

> This is in my lab. Plain passthrough setup with automap enabled on the F5. s3 > & curl work fine as far as queries go. But file transfer rate degrades badly > once I start file up/download. Maybe the difference can be attributed to LAN client traffic with jumbo frames vs F5 using a smaller WAN

Re: [ceph-users] PG Scaling

Thank you for your response. The number of replicas is already set to 3. So if I simply increase the number of pg’s they will also start to move or is that simply triggered with size alterations? I suppose since this will generate movement in the cluster network it is ideal to do this operation

Re: [ceph-users] if partition name changes, will ceph get corrupted?

> We use /dev/disk/by-path for this reason, but we confirmed that is stable > for our HBAs. Maybe /dev/disk/by-something is consistent with your > controller. The upstart/udev scripts will handle mounting and osd id detection, at least on Ubuntu. -- Kyle

[ceph-users] OSD down after PG increase

I've increased PG number to a running cluster. After this operation, all OSDs from one node was marked as down. Now, after a while, i'm seeing that OSDs are slowly coming up again (sequentially) after rebalancing. Is this an expected behaviour ? ___ cep

Re: [ceph-users] PG Scaling

Most things will cause data movement... If you are going to have different failure zones within your crush map, I would edit your crush map and define those failure zones/buckets, first. This will cause data movement when you inject the new crush map into the cluster. This will immediately ca

Re: [ceph-users] "no user info saved" after user creation / can't create buckets

The saga continues... So, after fiddling with haproxy a bit, I managed to make sure that my requests were hitting the RADOS Gateway. NOW, I get a 403 from my ruby script: 2014-03-12 23:34:08.289670 7fda9bfbf700 1 == starting new request req=0x215a780 = 2014-03-12 23:34:08.305105 7fda9bf

Re: [ceph-users] PG Scaling

Awesome thanks for the info. We have just began testing phase. I have 10Gig interfaces on both the cluster and public interfaces and using fast disks so I probably won’t feel much of a difference. Since this is just a test setup I have some freedom here but nice to know the consequences. Karol

Re: [ceph-users] "no user info saved" after user creation / can't create buckets

And the debug log because that last log was obviously not helpful... 2014-03-12 23:57:49.497780 7ff97e7dd700 1 == starting new request req=0x23bc650 = 2014-03-12 23:57:49.498198 7ff97e7dd700 2 req 1:0.000419::PUT /test::initializing 2014-03-12 23:57:49.498233 7ff97e7dd700 10 host=s3.amaz

Re: [ceph-users] "no user info saved" after user creation / can't create buckets

Increasing the logging further, and I notice the following: 2014-03-13 00:27:28.617100 7f6036ffd700 20 rgw_create_bucket returned ret=-1 bucket=test(@.rgw.buckets[us-west-1.15849318.1]) But hope that .rgw.buckets doesn't have to exist... and that @.rgw.buckets is perhaps telling of something? I

Re: [ceph-users] "no user info saved" after user creation / can't create buckets

And, I figured out the issue. The utility I was using to create pools, zones, and regions automatically failed to do two things: - create rgw.buckets and rgw.buckets.index for each zone - setup placement pools for each zone I did both of those, and now everything is working. Thanks, me, for the

[ceph-users] dashing ceph

Hi. Here's a go at a dashing dashboard for CEPH: https://github.com/rochaporto/dashing-ceph Quick and easy, but i think it looks nice for a first go. I'll probably add a couple more dashboards with additional data and get them rotating, the ceph tools are pretty helpful for that. And as we use

Re: [ceph-users] Some Questions about using ceph with VMware

On 3/12/2014 6:24 PM, Florian Krauß wrote: Hello everyone, this is the first time i ever write to a mailing list, please be patient with me (especially for my poor english)… Im trying to reach my Bachelors Degree in Computer Science, Im doing a Project which involv

[ceph-users] RGW python API failed

I use the RGW python API but never succeeded. test-ceph.py: #!/usr/bin/python # Filename: test-ceph.py import boto import boto.s3.connection access_key = 'TJLN5ID' secret_key = 'EcaCMS1\/4b3\/AAM50Q' conn = boto.connect_s3( aws_access_key_id = access_key,

Re: [ceph-users] Access Denied errors

Any ideas on this? Anything I can do to further troubleshoot? -Steve - Original Message - > From: "Steve Carter" > To: "Yehuda Sadeh" > Cc: ceph-users@lists.ceph.com > Sent: Tuesday, March 11, 2014 4:42:24 PM > Subject: Re: [ceph-users] Access Denied errors > Just to be complete, a

Re: [ceph-users] RGW python API failed

On Wed, Mar 12, 2014 at 7:40 PM, wsnote wrote: > I use the RGW python API but never succeeded. > > test-ceph.py: > #!/usr/bin/python > # Filename: test-ceph.py > import boto > import boto.s3.connection > access_key = 'TJLN5ID' > secret_key = 'EcaCMS1\/4b3\/AAM50Q' > > c

Re: [ceph-users] Access Denied errors

You need to try switching the calling format. Instead of accessing http://admin.XXX/user, try accessing http://XXX/admin/user. There's a good chance that the subdomain calling format does not work with the admin api. Yehuda On Wed, Mar 12, 2014 at 8:58 PM, Steve Carter wrote: > Any ideas on this

[ceph-users] Remove volume

Hi All, I have ceph installed on Ubuntu nodes. At present I have the following volumes in Volumes pool: - root@compute:/home/oss# rbd -p volumes ls volume-53f0dc29-956f-48f1-8db1-b0f9c1b0e9f1 volume-55abf0d4-01a7-41d0-9e7e-407ad0db213c volume-a73d1bd0-2937-41c4-bbca-2545454eefac volume-bd