Re: [ceph-users] Is it safe to increase pg numbers in a production environment

2015-08-05 Thread Haomai Wang
On Wed, Aug 5, 2015 at 1:36 PM, 乔建峰 wrote: > Add the mailing lists. > > 2015-08-05 13:34 GMT+08:00 乔建峰 : >> >> Hi Haomai, >> >> Thank you for the prompt response and the suggestion. >> >> I cannot agree with you more about using multiple pools in one flexible >> cluster. Per the scenario you descr

Re: [ceph-users] migrating cephfs metadata pool from spinning disk to SSD.

2015-08-05 Thread John Spray
On Tue, Aug 4, 2015 at 10:36 PM, Bob Ababurko wrote: > My writes are not going as I would expect wrt to IOPS(50-1000 IOPs) & write > throughput( ~25MB/s max). I'm interested in understanding what it takes to > create a SSD pool that I can then migrate the current Cephfs_metadata pool > to. I sus

Re: [ceph-users] CephFS vs Lustre performance

2015-08-05 Thread jupiter
Thanks Mark and Scottix for the helpful comments. Cheers. - jupiter On Wed, Aug 5, 2015 at 3:06 AM, Scottix wrote: > I'll be more of a third-party person and try to be factual. =) > > I wouldn't throw off Gluster too fast yet. > Besides what you described with the object and disk storage. > It

[ceph-users] 160 Thousand ceph-client.admin.*.asok files : Wired problem , never seen before

2015-08-05 Thread Vickey Singh
Hello Community I am facing a very wired problem with Ceph socket files. For all monitor nodes under /var/run/ceph/ i can see ~160 Thousands asok files , most of the file names are ceph-client.admin.* *If i delete these files are the getting generated very quickly.* *Could someone please answ

Re: [ceph-users] Is it safe to increase pg number in a production environment

2015-08-05 Thread Jan Schermer
Hi, comments inline. > On 05 Aug 2015, at 05:45, Jevon Qiao wrote: > > Hi Jan, > > Thank you for the detailed suggestion. Please see my reply in-line. > On 5/8/15 01:23, Jan Schermer wrote: >> I think I wrote about my experience with this about 3 months ago, including >> what techniques I used

Re: [ceph-users] Is it safe to increase pg numbers in a production environment

2015-08-05 Thread Jan Schermer
Hi, is adjusting crush weight really a good solution for this? Crush weight out of the box corresponds to OSD capacity in TB and this looks like a good “weight” to me. The issue is not in a bucket having wrong weight, but somewhere else depending on CRUSH. We actually use “osd reweight” for this

[ceph-users] Unable to start libvirt VM when using cache tiering.

2015-08-05 Thread Pieter Koorts
Hi All, This seems to be a weird issue. Firstly all deployment is done with "ceph-deploy" and 3 host machines acting as MON and OSD using the Hammer release on Ubuntu 14.04.3 and running KVM (libvirt). When using vanilla CEPH, single rbd pool no log device or cache tiering, the virtual machin

Re: [ceph-users] Unable to start libvirt VM when using cache tiering.

2015-08-05 Thread Burkhard Linke
Hi, On 08/05/2015 02:13 PM, Pieter Koorts wrote: Hi All, This seems to be a weird issue. Firstly all deployment is done with "ceph-deploy" and 3 host machines acting as MON and OSD using the Hammer release on Ubuntu 14.04.3 and running KVM (libvirt). When using vanilla CEPH, single rbd pool

Re: [ceph-users] Unable to start libvirt VM when using cache tiering.

2015-08-05 Thread Pieter Koorts
Hi Burkhard, I seemed to have missed that part but even though allowing access (rwx) to the cache pool it still has a similar (not same) problem. The VM process starts but it looks more like a dead or stuck process trying forever to start and has high CPU (for the qemu-system-x86 process). Whe

Re: [ceph-users] Unable to start libvirt VM when using cache tiering.

2015-08-05 Thread Burkhard Linke
Hi, On 08/05/2015 02:54 PM, Pieter Koorts wrote: Hi Burkhard, I seemed to have missed that part but even though allowing access (rwx) to the cache pool it still has a similar (not same) problem. The VM process starts but it looks more like a dead or stuck process trying forever to start and

Re: [ceph-users] Unable to start libvirt VM when using cache tiering.

2015-08-05 Thread Pieter Koorts
Hi,This is my OSD dump below###osc-mgmt-1:~$ sudo ceph osd dump | grep poolpool 0 'rbd' replicated size 3 min_size 2 crush_ruleset 0 object_hash rjenkins pg_num 128 pgp_num 128 last_change 43 lfor 43 flags hashpspool tiers 1 read_tier 1 write_tier 1 stripe_width 0pool 1 'ssd' re

[ceph-users] Setting up a proper mirror system for Ceph

2015-08-05 Thread Wido den Hollander
Hi, One of the first things I want to do as the Ceph User Committee is set up a proper mirror system for Ceph. Currently there is ceph.com, eu.ceph.com and au.ceph.com (thanks Matthew!), but this isn't the way I want to see it. I want to set up a series of localized mirrors from there you can ea

Re: [ceph-users] Unable to start libvirt VM when using cache tiering.

2015-08-05 Thread Burkhard Linke
Hi, On 08/05/2015 03:09 PM, Pieter Koorts wrote: Hi, This is my OSD dump below ### osc-mgmt-1:~$ sudo ceph osd dump | grep pool pool 0 'rbd' replicated size 3 min_size 2 crush_ruleset 0 object_hash rjenkins pg_num 128 pgp_num 128 last_change 43 lfor 43 flags hashpspool t

Re: [ceph-users] Setting up a proper mirror system for Ceph

2015-08-05 Thread Jan Schermer
I remember when ceph.com was down a while ago - it hurts. Thank you for this. Cloudflare works and should be free for the website itself. Not sure how they handle caching of “larger” (not website) objects for repositories etc, might be plug and play or might require integration with their CDN. J

Re: [ceph-users] Ceph Design

2015-08-05 Thread Dominik Zalewski
I would suggest splitting OSDs across two or more SSD journals (depending on OSD write speed and SSD sustained speed limits) e.g 2x Intel S3700 400GB for 8-10 OSDs or 4x Intel S3500 300GB for 8-10 OSDs (it may vary depending on the setup) If you RAID-1 SSD journals they will potentially "wear out

Re: [ceph-users] Setting up a proper mirror system for Ceph

2015-08-05 Thread David Moreau Simard
Would love to be a part of this Wido, we currently have a mirror at ceph.mirror.iweb.ca based on the script you provided me a while back. It is already available over http, rsync, IPv4 and IPv6. The way we currently mirror it does feel a bit clunky and I would welcome a better way to mirror Cep

Re: [ceph-users] Ceph Design

2015-08-05 Thread SUNDAY A. OLUTAYO
I intend to have 5-8 OSDs for 400GB SSD. Should there be different partitions for each OSD on the SSD? Thanks, Sunday Olutayo - Original Message - From: "Dominik Zalewski" To: "SUNDAY A. OLUTAYO" , "ceph-users" Sent: Wednesday, August 5, 2015 3:38:20 PM Subject: Re: [ceph

[ceph-users] HAproxy for RADOSGW

2015-08-05 Thread Ray Sun
.econe.com node-8d80.econe.com:80 check cookie node-8d80.econe.com -- DEBUG: signature-v4 headers: {'x-amz-content-sha256': 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': 'AWS4-HMAC-SHA256 Credential=YUY

Re: [ceph-users] Ceph Design

2015-08-05 Thread Dominik Zalewski
Yes, there should a separate partition per OSD. You are probably looking at 10-20GB journal partition per OSD. If you are creating your cluster using ceph-deploy it can create journal partitions for you. "The expected throughput number should include the expected disk throughput (i.e., sustained d

Re: [ceph-users] Unable to start libvirt VM when using cache tiering.

2015-08-05 Thread Pieter Koorts
Hi I suspect something more sinister may be going on. I have set the values (though smaller) on my cluster but the same issue happens. I also find when the VM is trying to start there might be an IRQ flood as processes like ksoftirqd seem to use more CPU than they should.

Re: [ceph-users] Ceph Design

2015-08-05 Thread SUNDAY A. OLUTAYO
I appreciate your help, I will use ceph-deploy and I will start with 10GB journal size. Thanks, Sunday Olutayo - Original Message - From: "Dominik Zalewski" To: "SUNDAY A. OLUTAYO" , "ceph-users" Sent: Wednesday, August 5, 2015 4:48:24 PM Subject: Re: [ceph-users] Ceph Design

Re: [ceph-users] Is it safe to increase pg number in a production environment

2015-08-05 Thread Marek Dohojda
I started with 7 and expended it to 14 with starting PG of 512 to 4096, as recommended. Unfortunately I can’t tell you the exact IO impact as I’ve done my changes in the off hours where the impact wasn’t important, I could see reduction in performance but since it had no impact on me I didn’t

Re: [ceph-users] Unable to start libvirt VM when using cache tiering.

2015-08-05 Thread Burkhard Linke
Hi, On 08/05/2015 05:54 PM, Pieter Koorts wrote: Hi I suspect something more sinister may be going on. I have set the values (though smaller) on my cluster but the same issue happens. I also find when the VM is trying to start there might be an IRQ flood as processes like ksoftirqd seem to u

Re: [ceph-users] Ceph Design

2015-08-05 Thread Somnath Roy
FYI, if you supply a block device partition as journal, the param osd_journal_size is ignored, the entire partition is used by ceph. Thanks & Regards Somnath From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Dominik Zalewski Sent: Wednesday, August 05, 2015 8:48 AM To: SU

Re: [ceph-users] rados bench multiple clients error

2015-08-05 Thread Ivo Jimenez
Is this flag undocumented on purpose? What's the difference between --run-name and --name? On Mon, Aug 3, 2015 at 9:34 AM Sheldon Mustard wrote: > On Fri, Jul 31, 2015 at 11:11 AM, Kenneth Waegeman > wrote: > > This works when only using 1 host.. > > Is there a way to run the benchmarks with mu

[ceph-users] systemd-udevd: failed to execute '/usr/bin/ceph-rbdnamer'

2015-08-05 Thread Bruce McFarland
I've been asked to look at the performance of RHEL 7.1/RHCS 1.3. I keep running into these errors on 1 of my RHEL 7.1 client systems. The rbd devices are still present, but ceph-rbdname Is not in /usr/bin, but it is in trusty /usr/bin. Much like the rbdmap init script that ships with RHEL 7.1, b

Re: [ceph-users] migrating cephfs metadata pool from spinning disk to SSD.

2015-08-05 Thread Bob Ababurko
Hi John, You are correct in that my expectations may be incongruent with what is possible with ceph(fs). I'm currently copying many small files(images) from a netapp to the cluster...~35k sized files to be exact and the number of objects/files copied thus far is fairly significant(below in bold):

[ceph-users] [ANN] ceph-deploy 1.5.27 released

2015-08-05 Thread Travis Rhoden
Hi everyone, A new version of ceph-deploy has been released. Version 1.5.27 includes the following: - a new "ceph-deploy repo" command that allows for adding and removing custom repo definitions - Makes commands like "ceph-deploy install --rgw" only install the RGW component of Ceph. This work

Re: [ceph-users] migrating cephfs metadata pool from spinning disk to SSD.

2015-08-05 Thread Bob Ababurko
I have installed diamond(built by ksingh found at https://github.com/ksingh7/ceph-calamari-packages) on the MDS node and I am not seeing the mds_server.handle_client_request OR objecter.op_active metrics being sent to graphite. Mind you, this is not the graphite that is part of the calamari instal

[ceph-users] radosgw + civetweb latency issue on Hammer

2015-08-05 Thread Srikanth Madugundi
Hi, After upgrading to Hammer and moving from apache to civetweb. We started seeing high PUT latency in the order of 2 sec for every PUT request. The GET request lo Attaching the radosgw logs for a single request. The ceph.conf has the following configuration for civetweb. [client.radosgw.gatewa

Re: [ceph-users] [ANN] ceph-deploy 1.5.27 released

2015-08-05 Thread Travis Rhoden
Hi Nigel, On Wed, Aug 5, 2015 at 9:00 PM, Nigel Williams wrote: > On 6/08/2015 9:45 AM, Travis Rhoden wrote: >> >> A new version of ceph-deploy has been released. Version 1.5.27 >> includes the following: > > > Has the syntax for use of --zap-disk changed? I moved it around but it is no > longer

Re: [ceph-users] migrating cephfs metadata pool from spinning disk to SSD.

2015-08-05 Thread Bob Ababurko
I found a way to get the stats you mentioned: mds_server.handle_client_request & objecter.op_active. I can see these values when I run: ceph daemon mds. perf dump I recently restarted the mds server so my stats reset but I still have something to share: "mds_server.handle_client_request": 44060