Re: [ceph-users] RGW multisite - second cluster woes

2016-08-18 Thread Shilpa Manjarabad Jagannath
- Original Message - > From: "Ben Morrice" > To: ceph-users@lists.ceph.com > Sent: Thursday, August 18, 2016 8:59:30 PM > Subject: [ceph-users] RGW multisite - second cluster woes > > Hello, > > I am trying to configure a second cluster into an existing Jewel RGW > installation. > > I

Re: [ceph-users] Spreading deep-scrubbing load

2016-08-18 Thread Mark Kirkwood
On 19/08/16 17:33, Christian Balzer wrote: On Fri, 19 Aug 2016 15:39:13 +1200 Mark Kirkwood wrote: It would be cool to have a command or api to alter/set the last deep scrub timestamp - as it seems to me that the only way to change the distribution of deep scrubs is to perform deep scrubs...

[ceph-users] Using S3 java SDK to change a bucket acl fails. ceph version 10.2.2

2016-08-18 Thread zhu tong
Error is : Exception in thread "main" com.amazonaws.services.s3.model.AmazonS3Exception: Bad Request (Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request ID: null), S3 Extended Request ID: null I have tried it with multiple SDK version, some shows different description (

Re: [ceph-users] Understanding write performance

2016-08-18 Thread Christian Balzer
Hello, see below, inline. On Thu, 18 Aug 2016 21:41:33 -0700 lewis.geo...@innoscale.net wrote: > Hi Christian, > Thank you for the follow-up on this. > > I answered those questions inline below. > > Have a good day, > > Lewis George > > > --

Re: [ceph-users] Spreading deep-scrubbing load

2016-08-18 Thread Christian Balzer
Holly thread necromancy Batman! On Fri, 19 Aug 2016 15:39:13 +1200 Mark Kirkwood wrote: > On 15/06/16 13:18, Christian Balzer wrote: > > > > "osd_scrub_min_interval": "86400", > > "osd_scrub_max_interval": "604800", > > "osd_scrub_interval_randomize_ratio": "0.5", > > Latest Hamme

Re: [ceph-users] Understanding write performance

2016-08-18 Thread lewis.geo...@innoscale.net
Hi Christian, Thank you for the follow-up on this. I answered those questions inline below. Have a good day, Lewis George From: "Christian Balzer" Sent: Thursday, August 18, 2016 6:31 PM To: ceph-users@lists.ceph.com Cc: "lewis.geo...@inn

Re: [ceph-users] Spreading deep-scrubbing load

2016-08-18 Thread Mark Kirkwood
On 15/06/16 13:18, Christian Balzer wrote: "osd_scrub_min_interval": "86400", "osd_scrub_max_interval": "604800", "osd_scrub_interval_randomize_ratio": "0.5", Latest Hammer and afterwards can randomize things (spreading the load out), but if you want things to happen within a cert

Re: [ceph-users] Understading osd default min size

2016-08-18 Thread Christian Balzer
On Fri, 19 Aug 2016 01:56:13 + Erick Lazaro wrote: > Hi. > > I would like to understand how the parameter default OSD min size pool. Did you read this? http://docs.ceph.com/docs/hammer/rados/operations/pools/#set-the-number-of-object-replicas > For example, I set up: > > osd default pool si

[ceph-users] Fail to automount osd after reboot when the /var Partition is ext4 but success automount when /var Partition is xfs

2016-08-18 Thread Leo Yu
hi,cepher i have deploy a cluster jewel 10.2.2,and fail to automount osd after reboot when the /var Partition is ext4: [root@node1 ~]# lsblk -f NAME FSTYPE LABEL UUID MOUNTPOINT fd0 sda ├─sda1 ext4 497a4f82-3cbf-4e27-b026-cdd3c5ecc2dd /boot └─sda2

[ceph-users] Understading osd default min size

2016-08-18 Thread Erick Lazaro
Hi. I would like to understand how the parameter default OSD min size pool. For example, I set up: osd default pool size = 3 osd default min pool size = 2 Failing 1 OSD, the ceph will block writing in degraded pgs? ___ ceph-users mailing list ceph-user

[ceph-users] Fail to automount osd after reboot when the /var Partition is ext4 but success automount when /var Partition is ext4

2016-08-18 Thread Leo Yu
hi,cepher i have deploy a cluster jewel 10.2.2,and fail to automount osd after reboot with the system Partition: [root@node1 ~]# lsblk -f NAME FSTYPE LABEL UUID MOUNTPOINT fd0 sda ├─sda1 ext4 497a4f82-3cbf-4e27-b026-cdd3c5ecc2dd /boot └─sda2 LVM2_

Re: [ceph-users] Understanding write performance

2016-08-18 Thread Christian Balzer
Hello, On Thu, 18 Aug 2016 12:03:36 -0700 lewis.geo...@innoscale.net wrote: > Hi, > So, I have really been trying to find information about this without > annoying the list, but I just can't seem to get any clear picture of it. I > was going to try to search the mailing list archive, but it s

Re: [ceph-users] Simple question about primary-affinity

2016-08-18 Thread Christian Balzer
Hello, completely ignoring your question about primary-affinity (which always struck me as a corner case thing). ^o^ If you're adding SSDs to your cluster you will want to: a) use them for OSD journals (if you're not doing so already) b) create dedicated pools for high speed data (i.e. RBD ima

Re: [ceph-users] CephFS Fuse ACLs

2016-08-18 Thread Brady Deetz
apparently fuse_default_permission and client_acl_type have to be in the fstab entry instead of the ceph.conf. Sorry for polluting the mailing list with an amateur mis-configuration. On Thu, Aug 18, 2016 at 4:26 PM, Brady Deetz wrote: > I'm having an issue with ACLs on my CephFS test environmen

[ceph-users] CephFS Fuse ACLs

2016-08-18 Thread Brady Deetz
I'm having an issue with ACLs on my CephFS test environment. Am I an idiot or is something weird going on? TLDR; I setfacl as root for a local user and the user still can't access the file. Example: root@test-client:/media/cephfs/storage/labs# touch test root@test-client:/media/cephfs/storage/lab

Re: [ceph-users] Rbd map command doesn't work

2016-08-18 Thread EP Komarla
I changed the profile to Hammer and it works. This bring up a question, by changing the profile to “Hammer” am I going to lose some of the performance optimizations done in ‘Jewel’? - epk From: Bruce McFarland [mailto:bkmcfarl...@earthlink.net] Sent: Tuesday, August 16, 2016 4:52 PM To: Somnat

Re: [ceph-users] How can we repair OSD leveldb?

2016-08-18 Thread Sean Sullivan
We have a hammer cluster that experienced a similar power failure and ended up corrupting our monitors leveldb stores. I am still trying to repair ours but I can give you a few tips that seem to help. 1.) I would copy the database off to somewhere safe right away. Just opening it seems to change

[ceph-users] Understanding write performance

2016-08-18 Thread lewis.geo...@innoscale.net
Hi, So, I have really been trying to find information about this without annoying the list, but I just can't seem to get any clear picture of it. I was going to try to search the mailing list archive, but it seems there is an error when trying to search it right now(posting below, and sending t

Re: [ceph-users] Signature V2

2016-08-18 Thread jan hugo prins
I have been able to reproduce the error and create a debug log from the failure. I can't post the debug log here because there is sensitive information in the debug log like access keys etc. Where can I send this log for analysis? And who is able to have a look at this? A small part of the debug lo

Re: [ceph-users] [Scst-devel] Thin Provisioning and Ceph RBD's

2016-08-18 Thread Alex Gorbachev
On Sat, Aug 13, 2016 at 4:51 PM, Alex Gorbachev wrote: > On Sat, Aug 13, 2016 at 12:36 PM, Alex Gorbachev > wrote: >> On Mon, Aug 8, 2016 at 7:56 AM, Ilya Dryomov wrote: >>> On Sun, Aug 7, 2016 at 7:57 PM, Alex Gorbachev >>> wrote: > I'm confused. How can a 4M discard not free anything?

Re: [ceph-users] Designing ceph cluster

2016-08-18 Thread Vasu Kulkarni
Also most of the terminology looks like from Openstack and SAN, Here are the right terminology that should be used for Ceph http://docs.ceph.com/docs/master/glossary/ On Thu, Aug 18, 2016 at 8:57 AM, Gaurav Goyal wrote: > Hello Mart, > > My Apologies for that! > > We are couple of office colleag

Re: [ceph-users] Designing ceph cluster

2016-08-18 Thread Peter Hinman
If you are wanting to run VMs, OSD, and Monitors all on the same hardware in a lab environment, it sounds like Proxmox might simplify things for you. Peter On 8/18/2016 9:57 AM, Gaurav Goyal wrote: Hello Mart, My Apologies for that! We are couple of office colleagues using the common gmail

Re: [ceph-users] Designing ceph cluster

2016-08-18 Thread Gaurav Goyal
Hello Mart, My Apologies for that! We are couple of office colleagues using the common gmail account. That has caused the nuisance. Thanks for your response! On Thu, Aug 18, 2016 at 6:00 AM, Mart van Santen wrote: > Dear Guarav, > > Please respect everyones time & timezone differences. Floodi

Re: [ceph-users] Reading payload from rados_watchcb2_t callback

2016-08-18 Thread LOPEZ Jean-Charles
Hi Nick, a good read to see what’s in it. http://dachary.org/?p=1904 JC > On Aug 18, 2016, at 08:28, Nick Fisk wrote: > > Just to answer myself in case anyone stumbles across this in the future. I > was on the right track, but I think there are null > characters before the text payload which

[ceph-users] RGW multisite - second cluster woes

2016-08-18 Thread Ben Morrice
Hello, I am trying to configure a second cluster into an existing Jewel RGW installation. I do not get the expected output when I perform a 'radosgw-admin realm pull'. My realm on the first cluster is called 'gold', however when doing a realm pull it doesn't reflect the 'gold' name or id and I ge

Re: [ceph-users] Reading payload from rados_watchcb2_t callback

2016-08-18 Thread Nick Fisk
Just to answer myself in case anyone stumbles across this in the future. I was on the right track, but I think there are null characters before the text payload which was tricking printf. In the end I managed to work it out and came up with this: char *temp = (char*)data+4; Which skips the 1st

Re: [ceph-users] Signature V2

2016-08-18 Thread jan hugo prins
I believe the same, but when you use V4 from s3cmd or the AWS S3 java API you get intermittend signature errors. Only after returning to V2 those errors are gone. Jan Hugo On 08/18/2016 03:51 PM, Chris Jones wrote: > I believe RGW Hammer and below use V2 and Jewel and above use V4. > > Thanks >

Re: [ceph-users] Signature V2

2016-08-18 Thread Chris Jones
I believe RGW Hammer and below use V2 and Jewel and above use V4. Thanks On Thu, Aug 18, 2016 at 7:32 AM, jan hugo prins wrote: > did some more searching and according to some info I found RGW should > support V4 signatures. > > http://tracker.ceph.com/issues/10333 > http://tracker.ceph.com/iss

Re: [ceph-users] Ceph all NVME Cluster sequential read speed

2016-08-18 Thread Nick Fisk
> -Original Message- > From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of nick > Sent: 18 August 2016 14:02 > To: n...@fisk.me.uk > Cc: 'ceph-users' > Subject: Re: [ceph-users] Ceph all NVME Cluster sequential read speed > > Thanks for the explanation. I thought th

Re: [ceph-users] Ceph all NVME Cluster sequential read speed

2016-08-18 Thread nick
Thanks for the explanation. I thought that when using a striped image 4MB of written data will be placed in 4 objects (with 4MB object size and when using 1MB of stripe unit and a count of 4). With that a single read of 4MB will hit 4 objects which might be in different PGs. So the read speed sh

Re: [ceph-users] Ceph all NVME Cluster sequential read speed

2016-08-18 Thread Nick Fisk
> -Original Message- > From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of nick > Sent: 18 August 2016 12:39 > To: n...@fisk.me.uk > Cc: 'ceph-users' > Subject: Re: [ceph-users] Ceph all NVME Cluster sequential read speed > > So after disabling logging and setting int

Re: [ceph-users] Ceph all NVME Cluster sequential read speed

2016-08-18 Thread nick
So after disabling logging and setting intel_idle.max_cstate=1 we reach 1953 IOPS for 4k blocksizes (with an iodepth of 1) instead of 1382. This is an increase of 41%. Very cool. Furthermore I played a bit with striping in RBD images. When choosing a 1MB stripe unit and a stripe count of 4 ther

Re: [ceph-users] Signature V2

2016-08-18 Thread jan hugo prins
did some more searching and according to some info I found RGW should support V4 signatures. http://tracker.ceph.com/issues/10333 http://tracker.ceph.com/issues/11858 The fact that everyone still modifies s3cmd to use Version 2 Signatures suggests to me that we have a bug in this code. If I use

[ceph-users] Signature V2

2016-08-18 Thread jan hugo prins
Hi everyone. To connect to my S3 gateways using s3cmd I had to set the option signature_v2 in my s3cfg to true. If I didn't do that I would get Signature mismatch errors and this seems to be because Amazon uses Signature version 4 while the S3 gateway of Ceph only supports Signature Version 2. No

Re: [ceph-users] Designing ceph cluster

2016-08-18 Thread Mart van Santen
Dear Guarav, Please respect everyones time & timezone differences. Flooding the mail-list won't help see below, On 08/18/2016 01:39 AM, Gaurav Goyal wrote: > Dear Ceph Users, > > Awaiting some suggestion please! > > > > On Wed, Aug 17, 2016 at 11:15 AM, Gaurav Goyal > mailto:er.gauravgo...@gma

Re: [ceph-users] Ceph all NVME Cluster sequential read speed

2016-08-18 Thread nick
Thanks for all the answers, we will disable logging and check the c-state CPU pinning. I did not hear so far of async messenger. After checking the mailing list it looks like one can test with ms_type = async option. I did not find the documentation for that (looks like this is a quite recent ad

Re: [ceph-users] Ceph all NVME Cluster sequential read speed

2016-08-18 Thread Nick Fisk
> -Original Message- > From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of > w...@42on.com > Sent: 18 August 2016 09:35 > To: nick > Cc: ceph-users > Subject: Re: [ceph-users] Ceph all NVME Cluster sequential read speed > > > > > Op 18 aug. 2016 om 10:15 heeft nic

[ceph-users] radosgw error in its log rgw_bucket_sync_user_stats()

2016-08-18 Thread zhu tong
Hi all, Version: 0.94.7 radosgw has reported the following error: 2016-08-16 15:26:06.883957 7fc2f0bfe700 0 ERROR: rgw_bucket_sync_user_stats() for user=user1, bucket=2537e61b32ca783432138237f234e610d1ee186e(@{i=.rgw.buckets.index,e=.rgw.buckets.extra}.rgw.buckets[default.4151.167]) returned

Re: [ceph-users] Ceph all NVME Cluster sequential read speed

2016-08-18 Thread w...@42on.com
> Op 18 aug. 2016 om 10:15 heeft nick het volgende geschreven: > > Hi, > we are currently building a new ceph cluster with only NVME devices. One Node > consists of 4x Intel P3600 2TB devices. Journal and filestore are on the same > device. Each server has a 10 core CPU and uses 10 GBit ether

[ceph-users] Ceph all NVME Cluster sequential read speed

2016-08-18 Thread nick
Hi, we are currently building a new ceph cluster with only NVME devices. One Node consists of 4x Intel P3600 2TB devices. Journal and filestore are on the same device. Each server has a 10 core CPU and uses 10 GBit ethernet NICs for public and ceph storage traffic. We are currently testing with

Re: [ceph-users] How can we repair OSD leveldb?

2016-08-18 Thread Wido den Hollander
> Op 17 augustus 2016 om 23:54 schreef Dan Jakubiec : > > > Hi Wido, > > Thank you for the response: > > > On Aug 17, 2016, at 16:25, Wido den Hollander wrote: > > > > > >> Op 17 augustus 2016 om 17:44 schreef Dan Jakubiec : > >> > >> > >> Hello, we have a Ceph cluster with 8 OSD that rec

[ceph-users] Merging CephFS data pools

2016-08-18 Thread Burkhard Linke
Hi, the current setup for CephFS at our site uses two data pools due to different requirements in the past. I want to merge these two pools now, eliminating the second pool completely. I've written a small script to locate all files on the second pool using their file layout attributes and r