[ceph-users] Speeding Up "rbd ls -l " output

Hi; I am using Hammer 0.49.9 release on my Ceph Storage, today i noticed that listing an rbd pool takes to much time then the old days. If i have more rbd images on pool it takes much more time. My clusters health is ok and currently there is no load on the cluster. Only rbd images are used to ser

[ceph-users] Migrating data from a Ceph clusters to another

Hi, I have 2 Ceph clusters, cluster A and cluster B. I want to move all the pools on A to B. The pool names don't conflict between clusters. I guess it's like RBD mirroring, except that it's pool mirroring. Is there any proper ways to do it? Thanks for any suggestions. Best, John Lin ___

Re: [ceph-users] Migrating data from a Ceph clusters to another

Hi. I recommend using rbd import/export. С уважением, Фасихов Ирек Нургаязович Моб.: +79229045757 2017-02-09 11:13 GMT+03:00 林自均 : > Hi, > > I have 2 Ceph clusters, cluster A and cluster B. I want to move all the > pools on A to B. The pool names don't conflict between clusters. I guess > it's l

Re: [ceph-users] Speeding Up "rbd ls -l " output

> Op 9 februari 2017 om 9:13 schreef Özhan Rüzgar Karaman > : > > > Hi; > I am using Hammer 0.49.9 release on my Ceph Storage, today i noticed that > listing an rbd pool takes to much time then the old days. If i have more > rbd images on pool it takes much more time. > It is the -l flag that

Re: [ceph-users] Migrating data from a Ceph clusters to another

Hi John, rbd mirroring can configured by pool.http://docs.ceph.com/docs/master/rbd/rbd-mirroring/ However the rbd mirroring method can only be used on rbd with layering feature, it can not mirror objects other than rbd for you. Sincerely, Craig Chi On 2017-02-09 16:24, Irek Fasikhovwrote: > Hi

Re: [ceph-users] Speeding Up "rbd ls -l " output

Hi Wido; Thanks for fast response rbd ls -l reads all images header for its sizes yes it makes sense you are right. My main problem is when i refresh a rbd storage pool using virsh over kvm(Ubuntu 14.04.5) it takes too much time then the old days and i suspect that virsh makes "rbd ls -l" over Cep

Re: [ceph-users] would people mind a slow osd restart during luminous upgrade?

On 17-02-09 05:09, Sage Weil wrote: Hello, ceph operators... Several times in the past we've had to do some ondisk format conversion during upgrade which mean that the first time the ceph-osd daemon started after upgrade it had to spend a few minutes fixing up it's ondisk files. We haven't had t

Re: [ceph-users] Migrating data from a Ceph clusters to another

Hi, Sorry I gave the wrong feature. rbd mirroring method can only be used on rbd with "journaling" feature (not layering). Sincerely, Craig Chi On 2017-02-09 16:41, Craig Chiwrote: > Hi John, > > rbd mirroring can configured by > pool.http://docs.ceph.com/docs/master/rbd/rbd-mirroring/ > Ho

Re: [ceph-users] Speeding Up "rbd ls -l " output

> Op 9 februari 2017 om 9:41 schreef Özhan Rüzgar Karaman > : > > > Hi Wido; > Thanks for fast response rbd ls -l reads all images header for its sizes > yes it makes sense you are right. > > My main problem is when i refresh a rbd storage pool using virsh over > kvm(Ubuntu 14.04.5) it takes t

Re: [ceph-users] Speeding Up "rbd ls -l " output

Thanks Wido, you are the best :) On Thu, Feb 9, 2017 at 11:50 AM, Wido den Hollander wrote: > > > Op 9 februari 2017 om 9:41 schreef Özhan Rüzgar Karaman < > oruzgarkara...@gmail.com>: > > > > > > Hi Wido; > > Thanks for fast response rbd ls -l reads all images header for its sizes > > yes it ma

Re: [ceph-users] ceph-mon memory issue jewel 10.2.5 kernel 4.4

Hi Jim, On 02/08/2017 07:45 PM, Jim Kilborn wrote: I have had two ceph monitor nodes generate swap space alerts this week. Looking at the memory, I see ceph-mon using a lot of memory and most of the swap space. My ceph nodes have 128GB mem, with 2GB swap (I know the memory/swap ratio is odd)

Re: [ceph-users] Migrating data from a Ceph clusters to another

Hi Irek & Craig, Sorry, I misunderstood "RBD mirroring". What I want to do is not like that. I just want to move all the data from a cluster to another. It can be achieved by `rados -p get ` for all objects on cluster A, and then `rados -p put ` on cluster B. Is there any tool for that? Best

Re: [ceph-users] would people mind a slow osd restart during luminous upgrade?

On 02/09/2017 04:19 AM, David Turner wrote: The only issue I can think of is if there isn't a version of the clients fully tested to work with a partially upgraded cluster or a documented incompatibility requiring downtime. We've had upgrades where we had to upgrade clients first and others that

Re: [ceph-users] would people mind a slow osd restart during luminous upgrade?

On Thu, Feb 09, 2017 at 10:41:44AM +0200, Henrik Korkuc wrote: > On 17-02-09 05:09, Sage Weil wrote: > >Does this concern anyone? It probably means the upgrades will take longer > >if you're going host by host since the time per host will go up. > In my opinion if this is clearly communicated (rel

Re: [ceph-users] ceph-mon memory issue jewel 10.2.5 kernel 4.4

Joao, Here is the information requested. Thanks for taking a look. Note that the below is after I restarted the ceph-mon processes yesterday. If this is not acceptable, I will have to wait until the issue reappears. This is on a small cluster. 4 ceph nodes, and 6 ceph kernel clients running ove

Re: [ceph-users] would people mind a slow osd restart during luminous upgrade?

Hi Sage, Is the update running in parallel for all OSDs being restarted? Because 5 min per server is different than 150 min when there are 30 OSDs there.. Thank you, George > On Feb 8, 2017, at 22:09, Sage Weil wrote: > > Hello, ceph operators... > > Several times in the past we've had to

Re: [ceph-users] would people mind a slow osd restart during luminous upgrade?

On Thu, 9 Feb 2017, George Mihaiescu wrote: > Hi Sage, > > Is the update running in parallel for all OSDs being restarted? > > Because 5 min per server is different than 150 min when there are 30 > OSDs there.. In parallel. sage > > Thank you, > George > > > On Feb 8, 2017, at 22:09, Sage

Re: [ceph-users] would people mind a slow osd restart during luminous upgrade?

On Thu, 9 Feb 2017, David Turner wrote: > The only issue I can think of is if there isn't a version of the clients > fully tested to work with a partially upgraded cluster or a documented > incompatibility requiring downtime. We've had upgrades where we had to > upgrade clients first and others tha

[ceph-users] Ceph security hardening

Hay All Does any one have an advise on hardening my ceph cluster? I have or ready doen the cephx auth part, but not sure if i can say limit my ceph user sudo permission to use only ceph commands. Any advise on this would be grateful ___ ceph-users mail

[ceph-users] Fwd: Ceph security hardening

Hay All Does any one have an advise on hardening my ceph cluster? I have or ready doen the cephx auth part, but not sure if i can say limit my ceph user sudo permission to use only ceph commands. Any advise on this would be grateful ___ ceph-users mail

Re: [ceph-users] would people mind a slow osd restart during luminous upgrade?

When we upgraded to Jewel 10.2.3 from Hammer 0.94.7 in our QA cluster we had issues with client incompatibility. We first tried upgrading our clients before upgrading the cluster. This broke creating RBDs, cloning RBDs, and probably many other things. We quickly called that test a wash and redep

Re: [ceph-users] Fwd: Ceph security hardening

You can change your ceph.conf file's permissions to only be readable by root so that you have to use sudo to run ceph commands. Configuring sudo to only work with certain commands is a simple and common practice which should be easy to implement. On Thu, Feb 9, 2017 at 10:12 AM nigel davies wrot

Re: [ceph-users] ceph-mon memory issue jewel 10.2.5 kernel 4.4

I've been trying to figure out the same thing recently - I had the same issues as others with jewel 10.2.3 (?) but for my current problem I don't think it's a ceph issue. Specifically ever since our last maintenance day, some of our OSD nodes having been suffering OSDs killed by OOM killer des

Re: [ceph-users] would people mind a slow osd restart during luminous upgrade?

On Thu, Feb 9, 2017 at 9:12 AM, David Turner wrote: > When we upgraded to Jewel 10.2.3 from Hammer 0.94.7 in our QA cluster we > had issues with client incompatibility. We first tried upgrading our > clients before upgrading the cluster. This broke creating RBDs, cloning > RBDs, and probably ma

Re: [ceph-users] PG stuck peering after host reboot

OK, I've had a look. Haven't been able to take a proper look at the network yet but here's what I've gathered on other fronts so far: * Marking either osd.595 or osd.7 out results in this: $ ceph health detail | grep -v stuck | grep 1.323 pg 1.323 is remapped+peering, acting [2147483647,1391,2

[ceph-users] Erasure Profile Update

Hi, I'm looking for the way to do erasure profile update regarding to nodes addition. Let's say on the first I have 5 OSD nodes with 3+2 erasure profile so all chunks, including the code chunks, will be spread on every OSD nodes. In the future, let's say I add 2 OSD nodes and I wan to have 5+2 era

[ceph-users] Radosgw scaling recommendation?

Hi all, We're doing some stress testing with clients hitting our rados gw nodes with simultaneous connections. When the number of client connections exceeds about 5400 we start seeing 403 forbidden errors and log messages like the following: 2017-02-09 08:53:16.915536 7f8c667bc700 0 NOTICE: requ

Re: [ceph-users] Radosgw scaling recommendation?

I'm not really an RGW expert, but I'd suggest increasing the "rgw_thread_pool_size" option to something much higher than the default 100 threads if you haven't already. RGW requires at least 1 thread per client connection, so with many concurrent connections some of them might end up timing ou

Re: [ceph-users] Erasure Profile Update

The erasure profile cannot be changed on a pool. If you want to change the profile to be 5+2 instead of 3+2, then you need to create a new pool with the new profile and migrate your data to it. [cid:imaged817a3.JPG@663a9504.4baf2b32]

Re: [ceph-users] ceph-mon memory issue jewel 10.2.5 kernel 4.4

Graham, I don’t think this is the issue I’m seeing. I’m running Centos on kernel 4.4.24-1. My processes aren’t dying. I have two clusters with 3 mons in each cluster. Over the last 3 months that the clusters have been running, this is only happened on two nodes, and only once per node. If

[ceph-users] CephFS root squash?

Does cephfs have an option for root squash, like nfs mounts do? I am trying to figure out how to allow my users to have sudo on their workstation, but not have that root access to the ceph kernel mounted volume. Can’t seem to find anything. Using cephx for the mount, but can’t find a “root squas

Re: [ceph-users] ceph-mon memory issue jewel 10.2.5 kernel 4.4

Hi Jim, I've got a few questions for you as it looks like we have a similar cluster for our ceph infrastructure. A quick overview of what we have. We are also running a small cluster of 3 storage nodes (30 osds in total) and 5 clients over 40gig/s infiniband link (ipoib). Ever since installin

Re: [ceph-users] osd_snap_trim_sleep keeps locks PG during sleep?

Ok, https://github.com/athanatos/ceph/tree/wip-snap-trim-sleep (based on master) passed a rados suite. It adds a configurable limit to the number of pgs which can be trimming on any OSD (default: 2). PGs trimming will be in snaptrim state, PGs waiting to trim will be in snaptrim_wait state. I su

Re: [ceph-users] osd_snap_trim_sleep keeps locks PG during sleep?

Building now From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Samuel Just Sent: 09 February 2017 19:22 To: Nick Fisk Cc: ceph-users@lists.ceph.com Subject: Re: [ceph-users] osd_snap_trim_sleep keeps locks PG during sleep? Ok, https://github.com/athanatos/ceph/tree/

[ceph-users] OSDs stuck unclean

We have 4 OSDs in test environment that are all stuck unclean I've tried rebuilding the whole environment with the same result. OSDs are running on XFS disk, partition 1 is OSD, partition 2 is journal Also seeing degraded despite having 4 OSDs and a default osd pool of 2

Re: [ceph-users] Radosgw scaling recommendation?

> Op 9 februari 2017 om 19:34 schreef Mark Nelson : > > > I'm not really an RGW expert, but I'd suggest increasing the > "rgw_thread_pool_size" option to something much higher than the default > 100 threads if you haven't already. RGW requires at least 1 thread per > client connection, so wi

Re: [ceph-users] OSDs stuck unclean

4 OSD nodes or daemons? please: * ceph -v * ceph -s * ceph osd tree On Fri, Feb 10, 2017 at 5:26 AM, Craig Read wrote: > We have 4 OSDs in test environment that are all stuck unclean > > > > I’ve tried rebuilding the whole environment with the same result. > > > > OSDs are running on XFS di

Re: [ceph-users] Radosgw scaling recommendation?

I'm curious how does the num_threads option to civetweb relate to the 'rgw thread pool size'? Should i make them equal? ie: rgw frontends = civetweb enable_keep_alive=yes port=80 num_threads=125 error_log_file=/var/log/ceph/civetweb.error.log access_log_file=/var/log/ceph/civetweb.access.log -

Re: [ceph-users] CephFS root squash?

On Thu, Feb 9, 2017 at 11:11 AM, Jim Kilborn wrote: > Does cephfs have an option for root squash, like nfs mounts do? > I am trying to figure out how to allow my users to have sudo on their > workstation, but not have that root access to the ceph kernel mounted volume. > > Can’t seem to find anyt

Re: [ceph-users] OSDs stuck unclean

Please provide us with crushmap * sudo ceph osd getcrushmap -o crushmap.`date +%Y%m%d%H` On Fri, Feb 10, 2017 at 5:46 AM, Craig Read wrote: > Sorry, 2 nodes, 6 daemons (forgot I added 2 daemons to see if it made a > difference) > > On CentOS7 > > Ceph -v: > > 10.2.5 > > Ceph -s: > > Health HEA

[ceph-users] RadosGW: No caching when S3 tokens are validated against Keystone?

We're using the Hammer version of RadosGW, with Keystone for authN/Z. When a user started sending a lot of S3 requests (using rclone), the load on our Keystone service has skyrocketed. This surprised me because all those requests are from the same user, and RadosGW has caching for Keystone tokens.

[ceph-users] trying to test S3 bucket lifecycles in Kraken

Hey, I am trying to do some testing of S3 bucket lifecycles in Kraken but I am unable to setup a lifecycle. RGW always returns "501 Not Implemented“. >>> PUT /pdtest_expire_test?lifecycle HTTP/1.1 […] http://s3.amazonaws.com/doc/2006-03-01/";>testEnabled1 >>> <<< HTTP/1.1 501 Not Implemented [

[ceph-users] 2 of 3 monitors down and to recover

Hey, I tried to simulate the failure of 2 monitors including their monmap, and to bring them up in my testing cluster. The ceph version is 10.2.5, the OS is REHL7.2, and the testing cluster has 3 nodes with 3 monitors and 24 osds, each node has 1 monitor and 8 osds. So, I stopped the 2 (the mon.a

[ceph-users] I can't create new pool in my cluster.

The version I'm using is 0.94.9 And when I want to create a pool, It shows: Error EINVAL: error running crushmap through crushtool: (1) Operation not permitted What's wrong about this? ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.

[ceph-users] reference documents of cbt(ceph benchmarking tool)

Hi guys, So I was investigating on benchmark tool for ceph. CBT seems to be a good candidate. But the documents on github is limited.Regarding using this tool on existing cluster,is there any specific documents for the procedure. And I'm trying to use it on ubuntu 14.04.I don't know whether this

Re: [ceph-users] I can't create new pool in my cluster.

What did you exactly do? On Fri, Feb 10, 2017 at 11:48 AM, 周威 wrote: > The version I'm using is 0.94.9 > > And when I want to create a pool, It shows: > > Error EINVAL: error running crushmap through crushtool: (1) Operation > not permitted > > What's wrong about this? > _

Re: [ceph-users] I can't create new pool in my cluster.

# ceph osd pool create test 128 Error EINVAL: error running crushmap through crushtool: (1) Operation not permitted # rados mkpool test error creating pool test: (22) Invalid argument 2017-02-10 10:53 GMT+08:00 Shinobu Kinjo : > What did you exactly do? > > On Fri, Feb 10, 2017 at 11:48 AM, 周威 w

Re: [ceph-users] I can't create new pool in my cluster.

I can find some log in ceph-mon.log about this: > 2017-02-10 10:47:54.264026 7f6a6eff4700 0 mon.ceph-test2@1(peon) e9 > handle_command mon_command({"prefix": "osd pool create", "pg_num": 128, > "pool": "test"} v 0) v1 > 2017-02-10 10:47:54.264132 7f6a6eff4700 0 log_channel(audit) log [INF] :

Re: [ceph-users] 2 of 3 monitors down and to recover

hi taotao :) you can follow http://docs.ceph.com/docs/master/rados/operations/add-or-rm-mons/ , Remove the non-surviving or problematic monitors. For example ... remember to backup the surviving monitor data before any futher. 2017-02-10 9:52 GMT+08:00 何涛涛(云平台事业部) : > Hey, > > I tried to sim

[ceph-users] Shrink cache target_max_bytes

Hi Cephers, Long story short: I'd like to shrink our cache pool a little. Is it safe to just alter cache target_max_byte and wait for objects to get evicted? Anything to take into account? Thanks! Regards, Kees ___ ceph-users mailing list ceph-users@