Re: [ceph-users] Ceph pg active+clean+inconsistent

Would you be able to execute ``ceph pg ${PG ID} query`` against that particular PG? On Wed, Dec 21, 2016 at 11:44 PM, Andras Pataki wrote: > Yes, size = 3, and I have checked that all three replicas are the same zero > length object on the disk. I think some metadata info is mismatching what > t

Re: [ceph-users] Can't create bucket (ERROR: endpoints not configured for upstream zone)

FWIW, this is still required with Jewel 10.2.5. It sounded like it was finally fixed from the release notes, but i had the same issue. Fortunately Micha's steps are easy and fix it right up. In my case i didn't think i had any mixed RGWs - was planning to stop them all first - but i had forgotten

[ceph-users] radosgw setup issue

I am trying to setup radosgw on a ceph cluster, and I am seeing some issues where google is not helping. I hope some of the developers would be able to help here. I tried to create radosgw as mentioned here [0] on a jewel cluster. And it gives the following error in log file after starting rad

Re: [ceph-users] How exactly does rgw work?

Yes, this is common practice. Daniel On 12/22/2016 02:34 PM, Gerald Spencer wrote: Wonderful, just as I expected. Do folks normally have several RGW running on individual machines with a load balancer at larger scales? On Wed, Dec 21, 2016 at 8:22 AM, LOPEZ Jean-Charles mailto:jelo...@redhat.c

Re: [ceph-users] How exactly does rgw work?

Wonderful, just as I expected. Do folks normally have several RGW running on individual machines with a load balancer at larger scales? On Wed, Dec 21, 2016 at 8:22 AM, LOPEZ Jean-Charles wrote: > Hi Gerald, > > for the s3 and swift case, the clients are not accessing the ceph cluster. > They ar

Re: [ceph-users] Cephalocon Sponsorships Open

I / my group / our organization would be interested in discussing our deployment of Ceph and how we are using it, deploying it, future plans etc. This sounds like an exciting event. We look forward to hearing more details. On Thu, Dec 22, 2016 at 1:44 PM, Patrick McGarry wrote: > Hey cephers, >

[ceph-users] Cephalocon Sponsorships Open

Hey cephers, Just letting you know that we're opening the flood gates for sponsorship opportunities at Cephalocon next year (23-25 Aug 2017, Boston, MA). If you would be interested in sponsoring/exhibiting at our inaugural Ceph conference, please drop me a line. Thanks! -- Best Regards, Patri

Re: [ceph-users] rgw leaking data, orphan search loop

HI Maruis, On Thu, Dec 22, 2016 at 12:00 PM, Marius Vaitiekunas wrote: > On Thu, Dec 22, 2016 at 11:58 AM, Marius Vaitiekunas > wrote: >> >> Hi, >> >> 1) I've written before into mailing list, but one more time. We have big >> issues recently with rgw on jewel. because of leaked data - the rate

Re: [ceph-users] How can I debug "rbd list" hang?

From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Stéphane Klein Sent: 22 December 2016 17:10 To: n...@fisk.me.uk Cc: ceph-users Subject: Re: [ceph-users] How can I debug "rbd list" hang? 2016-12-22 18:07 GMT+01:00 Nick Fisk mailto:n...@fisk.me.uk>>: I think you have pro

Re: [ceph-users] How can I debug "rbd list" hang?

2016-12-22 18:07 GMT+01:00 Nick Fisk : > I think you have probably just answered your previous question. I would > guess pauserd and pausewr, pauses read and write IO, hence your command to > list is being blocked on reads. > > > How can I fix that? Where is the documentation about this two flags

Re: [ceph-users] What is pauserd and pausewr status?

> Op 22 december 2016 om 17:55 schreef Stéphane Klein > : > > > Hi, > > I have this status: > > bash-4.2# ceph status > cluster 7ecb6ebd-2e7a-44c3-bf0d-ff8d193e03ac > health HEALTH_WARN > pauserd,pausewr,sortbitwise,require_jewel_osds flag(s) set > monmap e1: 3 mons

Re: [ceph-users] How can I debug "rbd list" hang?

I think you have probably just answered your previous question. I would guess pauserd and pausewr, pauses read and write IO, hence your command to list is being blocked on reads. From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Stéphane Klein Sent: 22 December 2016 17

[ceph-users] How can I debug "rbd list" hang?

Hi, I have this status: root@ceph-mon-1:/home/vagrant# ceph status cluster 7ecb6ebd-2e7a-44c3-bf0d-ff8d193e03ac health HEALTH_WARN pauserd,pausewr,sortbitwise,require_jewel_osds flag(s) set monmap e1: 3 mons at {ceph-mon-1= 172.28.128.2:6789/0,ceph-mon-2=172.28.128.3:678

[ceph-users] What is pauserd and pausewr status?

Hi, I have this status: bash-4.2# ceph status cluster 7ecb6ebd-2e7a-44c3-bf0d-ff8d193e03ac health HEALTH_WARN pauserd,pausewr,sortbitwise,require_jewel_osds flag(s) set monmap e1: 3 mons at {ceph-mon-1= 172.28.128.2:6789/0,ceph-mon-2=172.28.128.3:6789/0,ceph-mon-3=172.28

Re: [ceph-users] Orphaned objects after deleting rbd images

On Wed, Dec 21, 2016 at 10:33 PM, Jason Dillaman wrote: > [moving to ceph-users ...] > > You should be able to use the rados CLI to list all the objects in > your pool, excluding all objects associated with known, valid image > ids: > > rados ls -p rbd | grep -vE "($(rados -p rbd ls | grep rbd_hea

[ceph-users] BlueStore with v11.1.0 Kraken

Hi guys, I'm building a first test cluster for homelab, and would like to start using BlueStore since data loss is not critical. However, there are obviously no official documentation for basic best usage online yet. My original layout was using 2x single Xeon nodes with 24 GB RAM each under Prox

Re: [ceph-users] Clone data inconsistency in hammer

On Thu, 22 Dec 2016, Bartłomiej Święcki wrote: > Hi, > > I have problems runnign Kraken tools on Hammer/Jewel cluster (official 11.1.0 > debs), > it asserts: > > /build/ceph-11.1.0/src/mon/MonMap.cc: In function 'void > MonMap::sanitize_mons(std::map, entity_addr_t>&)' > thread 7fffd37fe700 ti

Re: [ceph-users] Clone data inconsistency in hammer

Hi, I have problems runnign Kraken tools on Hammer/Jewel cluster (official 11.1.0 debs), it asserts: /build/ceph-11.1.0/src/mon/MonMap.cc: In function 'void MonMap::sanitize_mons(std::map, entity_addr_t>&)' thread 7fffd37fe700 time 2016-12-22 12:26:23.457058 /build/ceph-11.1.0/src/mon/MonMap.c

Re: [ceph-users] cannot commit period: period does not have a master zone of a master zonegroup

> Op 20 december 2016 om 18:06 schreef Orit Wasserman : > > > On Tue, Dec 20, 2016 at 5:39 PM, Wido den Hollander wrote: > > > >> Op 15 december 2016 om 17:10 schreef Orit Wasserman : > >> > >> > >> Hi Wido, > >> > >> This looks like you are hitting http://tracker.ceph.com/issues/17364 > >> The

Re: [ceph-users] If I shutdown 2 osd / 3, Ceph Cluster say 2 osd UP, why?

2016-12-22 12:30 GMT+01:00 Henrik Korkuc : > try waiting a little longer. Mon needs multiple down reports to take OSD > down. And as your cluster is very small there is small amount (1 in this > case) of OSDs to report that others are down. > > Why this limitation? because my rbd mount on ceph-cli

Re: [ceph-users] If I shutdown 2 osd / 3, Ceph Cluster say 2 osd UP, why?

On 16-12-22 13:26, Stéphane Klein wrote: Hi, I have: * 3 mon * 3 osd When I shutdown one osd, I work great: cluster 7ecb6ebd-2e7a-44c3-bf0d-ff8d193e03ac health HEALTH_WARN 43 pgs degraded 43 pgs stuck unclean 43 pgs undersized recovery

Re: [ceph-users] When I shutdown one osd node, where can I see the block movement?

On 16-12-22 13:20, Stéphane Klein wrote: 2016-12-22 12:18 GMT+01:00 Henrik Korkuc >: On 16-12-22 13:12, Stéphane Klein wrote: HEALTH_WARN 43 pgs degraded; 43 pgs stuck unclean; 43 pgs undersized; recovery 24/70 objects degraded (34.286%); too few PGs p

[ceph-users] If I shutdown 2 osd / 3, Ceph Cluster say 2 osd UP, why?

Hi, I have: * 3 mon * 3 osd When I shutdown one osd, I work great: cluster 7ecb6ebd-2e7a-44c3-bf0d-ff8d193e03ac health HEALTH_WARN 43 pgs degraded 43 pgs stuck unclean 43 pgs undersized recovery 24/70 objects degraded (34.286%)

Re: [ceph-users] When I shutdown one osd node, where can I see the block movement?

2016-12-22 12:18 GMT+01:00 Henrik Korkuc : > On 16-12-22 13:12, Stéphane Klein wrote: > > HEALTH_WARN 43 pgs degraded; 43 pgs stuck unclean; 43 pgs undersized; > recovery 24/70 objects degraded (34.286%); too few PGs per OSD (28 < min > 30); 1/3 in osds are down; > > it says 1/3 OSDs are down. By

Re: [ceph-users] When I shutdown one osd node, where can I see the block movement?

On 16-12-22 13:12, Stéphane Klein wrote: HEALTH_WARN 43 pgs degraded; 43 pgs stuck unclean; 43 pgs undersized; recovery 24/70 objects degraded (34.286%); too few PGs per OSD (28 < min 30); 1/3 in osds are down; it says 1/3 OSDs are down. By default Ceph pools are setup with size 3. If your se

Re: [ceph-users] When I shutdown one osd node, where can I see the block movement?

That's correct :) On 22/12/2016 12:12, Stéphane Klein wrote: > HEALTH_WARN 43 pgs degraded; 43 pgs stuck unclean; 43 pgs undersized; > recovery 24/70 objects degraded (34.286%); too few PGs per OSD (28 < min > 30); 1/3 in osds are down; > > Here Ceph say there are 24 objects to move? > > > > _

Re: [ceph-users] When I shutdown one osd node, where can I see the block movement?

HEALTH_WARN 43 pgs degraded; 43 pgs stuck unclean; 43 pgs undersized; recovery 24/70 objects degraded (34.286%); too few PGs per OSD (28 < min 30); 1/3 in osds are down; Here Ceph say there are 24 objects to move? ___ ceph-users mailing list ceph-users@l

Re: [ceph-users] When I shutdown one osd node, where can I see the block movement?

As always: ceph status On 22/12/2016 11:53, Stéphane Klein wrote: > Hi, > > When I shutdown one osd node, where can I see the block movement? > Where can I see percentage progression? > > Best regards, > Stéphane > > > > ___ > ceph-users mailing lis

[ceph-users] How can I ask to Ceph Cluster to move blocks now when osd is down?

Hi, How can I ask to Ceph Cluster to move blocks now when osd is down? Best regards, Stéphane -- Stéphane Klein blog: http://stephane-klein.info cv : http://cv.stephane-klein.info Twitter: http://twitter.com/klein_stephane ___ ceph-users mailing list

[ceph-users] When I shutdown one osd node, where can I see the block movement?

Hi, When I shutdown one osd node, where can I see the block movement? Where can I see percentage progression? Best regards, Stéphane -- Stéphane Klein blog: http://stephane-klein.info cv : http://cv.stephane-klein.info Twitter: http://twitter.com/klein_stephane _

Re: [ceph-users] mount /dev/rbd0 /mnt/image2 + rm Python-2.7.13 -rf => freeze

On Thu, Dec 22, 2016 at 8:32 AM, Stéphane Klein wrote: > > > 2016-12-21 23:39 GMT+01:00 Stéphane Klein : >> >> >> >> 2016-12-21 23:33 GMT+01:00 Ilya Dryomov : >>> >>> What if you boot ceph-client-3 with >512M memory, say 2G? >> >> >> Success ! > > > > It is possible to add a warning message in rbd

Re: [ceph-users] rgw leaking data, orphan search loop

On Thu, Dec 22, 2016 at 11:58 AM, Marius Vaitiekunas < mariusvaitieku...@gmail.com> wrote: > Hi, > > 1) I've written before into mailing list, but one more time. We have big > issues recently with rgw on jewel. because of leaked data - the rate is > about 50GB/hour. > > We've hitted these bugs: >

[ceph-users] rgw leaking data, orphan search loop

Hi, 1) I've written before into mailing list, but one more time. We have big issues recently with rgw on jewel. because of leaked data - the rate is about 50GB/hour. We've hitted these bugs: rgw: fix put_acls for objects starting and ending with underscore ( issue#17625

Re: [ceph-users] OSD will not start after heartbeatsuicide timeout, assert error from PGLog

Hi, I hit this a few weeks ago, here is the related tracker. You might want to update it to reflect your case and upload logs. http://tracker.ceph.com/issues/17916 Nick > -Original Message- > From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of > Trygve Vea > Sent:

Re: [ceph-users] Clone data inconsistency in hammer

Hi Jason, I'll test kraken tools since it happened on production, everything works there since the clone is flattened after being created and the production equivalent of "test" user can access the image only after it has been flattened. The issue happened when someone accidentally removed not-ye

Re: [ceph-users] mount /dev/rbd0 /mnt/image2 + rm Python-2.7.13 -rf => freeze

2016-12-21 23:33 GMT+01:00 Ilya Dryomov : > > What if you boot ceph-client-3 with >512M memory, say 2G? > > With: * 512 M memory => failed * 1000 M memory => failed * 1500 M memory => success ___ ceph-users mailing list ceph-users@lists.ceph.com http://