Hello, We have the same loop in our jobs in 2 clusters. Only one difference is that our cluster don't use erasure coding. The same cluster version - 10.2.2. Any ideas, what could be wrong? Maybe, we need to upgrade? :)
BR, On Thu, Oct 13, 2016 at 6:15 PM, Yoann Moulin <yoann.mou...@epfl.ch> wrote: > Hello, > > I run a cluster in jewel 10.2.2, I have deleted the last Bucket of a > radosGW pool to delete this pool and recreate it in EC (was replicate) > > Detail of the pool : > > > pool 36 'erasure.rgw.buckets.data' replicated size 3 min_size 2 > crush_ruleset 0 object_hash rjenkins pg_num 128 pgp_num 128 last_change > 31459 flags hashpspool stripe_width 0 > > > POOLS: > > NAME ID USED %USED MAX AVAIL > OBJECTS > > erasure.rgw.buckets.data 36 11838M 0 75013G > 4735 > > After the GC, I found lots of orphan objects still remain in the pool : > > > $ rados ls -p erasure.rgw.buckets.data | egrep -c "(multipart|shadow)" > > 4735 > > $ rados ls -p erasure.rgw.buckets.data | grep -c multipart > > 2368 > > $ rados ls -p erasure.rgw.buckets.data | grep -c shadow > > 2367 > > example : > > > c9724aff-5fa0-4dd9-b494-57bdb48fab4e.1371134.1__ > multipart_CC-MAIN-2016-40/segments/1474738660158.61/ > warc/CC-MAIN-20160924173740-00147-ip-10-143-35-109.ec2.internal.warc.gz.2~ > WezpbEQW1C9nskvtnyAteCVoO3D255Q.29 > > c9724aff-5fa0-4dd9-b494-57bdb48fab4e.1371134.1__ > multipart_CC-MAIN-2016-40/segments/1474738660158.61/ > warc/CC-MAIN-20160924173740-00147-ip-10-143-35-109.ec2.internal.warc.gz.2~ > WezpbEQW1C9nskvtnyAteCVoO3D255Q.61 > > c9724aff-5fa0-4dd9-b494-57bdb48fab4e.1371134.1__ > shadow_segments/1466783398869.97/wet/CC-MAIN-20160624154958- > 00194-ip-10-164-35-72.ec2.internal.warc.wet.gz.2~7ru9WPCLMf9Lpi__ > TP1NXuYwjSU7KQK.11_1 > > c9724aff-5fa0-4dd9-b494-57bdb48fab4e.1371134.1__ > shadow_crawl-data/CC-MAIN-2016-26/segments/1466783396147.66/wet/CC-MAIN- > 20160624154956-00071-ip-10-164-35-72.ec2.internal.warc.wet.gz.2~ > 7bKg6WEmNo23IQ6rd8oWF_vbaG0QAFR.6_1 > > c9724aff-5fa0-4dd9-b494-57bdb48fab4e.1371134.1__ > shadow_segments/1466783398516.82/wet/CC-MAIN-20160624154958- > 00172-ip-10-164-35-72.ec2.internal.warc.wet.gz.2~ap5QynCJTco_L7yK6bn4M_ > bnHBbBe64.14_1 > > c9724aff-5fa0-4dd9-b494-57bdb48fab4e.1371134.1__ > multipart_CC-MAIN-2016-40/segments/1474738662400.75/ > warc/CC-MAIN-20160924173742-00076-ip-10-143-35-109.ec2.internal.warc.gz.2~ > LEM4bpbbdiTu86rs3Ew_LFNN_oHg_m7.13 > > c9724aff-5fa0-4dd9-b494-57bdb48fab4e.1371134.1__shadow_CC-MAIN-2016-40/ > segments/1474738662400.75/warc/CC-MAIN-20160924173742- > 00033-ip-10-143-35-109.ec2.internal.warc.gz.2~ > FrN02NmencyDwXavvuzwqR8M8WnWNbH.8_1 > > c9724aff-5fa0-4dd9-b494-57bdb48fab4e.1371134.1__multipart_segments/ > 1466783395560.14/wet/CC-MAIN-20160624154955-00118-ip-10- > 164-35-72.ec2.internal.warc.wet.gz.2~GqyEUdSepIxGwPOXfKLSxtS8miWGASe.3 > > c9724aff-5fa0-4dd9-b494-57bdb48fab4e.1371134.1__multipart_segments/ > 1466783395346.6/wet/CC-MAIN-20160624154955-00083-ip-10- > 164-35-72.ec2.internal.warc.wet.gz.2~cTQ86ZEmOvxYD4BUI7zW37X-JcJeMgW.19 > > c9724aff-5fa0-4dd9-b494-57bdb48fab4e.1371134.1__ > multipart_CC-MAIN-2016-40/segments/1474738660158.61/ > warc/CC-MAIN-20160924173740-00147-ip-10-143-35-109.ec2.internal.warc.gz.2~ > WezpbEQW1C9nskvtnyAteCVoO3D255Q.62 > > c9724aff-5fa0-4dd9-b494-57bdb48fab4e.1371134.1__shadow_CC-MAIN-2016-40/ > segments/1474738662400.75/warc/CC-MAIN-20160924173742- > 00259-ip-10-143-35-109.ec2.internal.warc.gz.2~1b- > olF9koids0gqT9DsO0y1vAsTOasf.12_1 > > c9724aff-5fa0-4dd9-b494-57bdb48fab4e.1371134.1__shadow_CC-MAIN-2016-40/ > segments/1474738660338.16/warc/CC-MAIN-20160924173740- > 00067-ip-10-143-35-109.ec2.internal.warc.gz.2~ > JxuX8v0DmsSgAr3iprPBoHx6PoTKRi6.19_1 > > c9724aff-5fa0-4dd9-b494-57bdb48fab4e.1371134.1__multipart_segments/ > 1466783397864.87/wet/CC-MAIN-20160624154957-00110-ip-10- > 164-35-72.ec2.internal.warc.wet.gz.2~q2_hY5oSoBWaSZgxh0NdK8JvxmEySPB.29 > > c9724aff-5fa0-4dd9-b494-57bdb48fab4e.1371134.1__ > shadow_segments/1466783396949.33/wet/CC-MAIN-20160624154956- > 00000-ip-10-164-35-72.ec2.internal.warc.wet.gz.2~ > kUInFVpsWy23JFm9eWNPiFNKlXrjDQU.18_1 > > c9724aff-5fa0-4dd9-b494-57bdb48fab4e.1371134.1__ > multipart_CC-MAIN-2016-40/segments/1474738662400.75/ > warc/CC-MAIN-20160924173742-00076-ip-10-143-35-109.ec2.internal.warc.gz.2~ > LEM4bpbbdiTu86rs3Ew_LFNN_oHg_m7.36 > > firstly, Can I delete the pool even if there is orphan object in ? Should > I delete other metadata (index, data_extra) pools related to this pool > defined in the zone ? is there other data I should clean to be sure to no > have side effect by removing those objects by deleting the pool > instead of deleting them with radosgw-admin orphan ? > > for now, I have followed this doc to find and delete them : > > https://access.redhat.com/documentation/en/red-hat-ceph- > storage/1.3/single/object-gateway-guide-for-ubuntu/#finding_orphan_objects > > I have ran this command : > > > radosgw-admin --cluster cephprod orphans find > > --pool=erasure.rgw.buckets.data > --job-id=erasure > > but it stuck on a loop, is this a normal behavior ? > > example of the output I have for at least 2h : > > > storing 1 entries at orphan.scan.erasure.linked.2 > > storing 1 entries at orphan.scan.erasure.linked.5 > > storing 1 entries at orphan.scan.erasure.linked.9 > > storing 1 entries at orphan.scan.erasure.linked.19 > > storing 1 entries at orphan.scan.erasure.linked.25 > > storing 1 entries at orphan.scan.erasure.linked.40 > > storing 1 entries at orphan.scan.erasure.linked.43 > > storing 1 entries at orphan.scan.erasure.linked.47 > > storing 1 entries at orphan.scan.erasure.linked.56 > > storing 1 entries at orphan.scan.erasure.linked.63 > > storing 1 entries at orphan.scan.erasure.linked.9 > > storing 1 entries at orphan.scan.erasure.linked.25 > > storing 1 entries at orphan.scan.erasure.linked.40 > > storing 1 entries at orphan.scan.erasure.linked.56 > > storing 1 entries at orphan.scan.erasure.linked.2 > > storing 1 entries at orphan.scan.erasure.linked.5 > > storing 1 entries at orphan.scan.erasure.linked.9 > > storing 1 entries at orphan.scan.erasure.linked.19 > > storing 1 entries at orphan.scan.erasure.linked.25 > > storing 1 entries at orphan.scan.erasure.linked.40 > > storing 1 entries at orphan.scan.erasure.linked.43 > > storing 1 entries at orphan.scan.erasure.linked.47 > > storing 1 entries at orphan.scan.erasure.linked.56 > > storing 1 entries at orphan.scan.erasure.linked.63 > > storing 1 entries at orphan.scan.erasure.linked.2 > > storing 1 entries at orphan.scan.erasure.linked.5 > > storing 1 entries at orphan.scan.erasure.linked.9 > > storing 1 entries at orphan.scan.erasure.linked.19 > > storing 1 entries at orphan.s can.erasure.linked.25 > > storing 1 entries at orphan.scan.erasure.linked.40 > > storing 1 entries at orphan.scan.erasure.linked.43 > > storing 1 entries at orphan.scan.erasure.linked.47 > > storing 1 entries at orphan.scan.erasure.linked.56 > > storing 1 entries at orphan.scan.erasure.linked.63 > > storing 1 entries at orphan.scan.erasure.linked.9 > > storing 1 entries at orphan.scan.erasure.linked.25 > > storing 1 entries at orphan.scan.erasure.linked.40 > > storing 1 entries at orphan.scan.erasure.linked.56 > > storing 1 entries at orphan.scan.erasure.linked.2 > > storing 1 entries at orphan.scan.erasure.linked.5 > > storing 1 entries at orphan.scan.erasure.linked.9 > > storing 1 entries at orphan.scan.erasure.linked.19 > > storing 1 entries at orphan.scan.erasure.linked.25 > > storing 1 entries at orphan.scan.erasure.linked.40 > > storing 1 entries at orphan.scan.erasure.linked.43 > > storing 1 entries at orphan.scan.erasure.linked.47 > > storing 1 entries at orphan.scan.erasure.linked.56 > > storing 1 entries at orphan.scan.erasure.linked.63 > > storing 1 entries at orphan.scan.erasure.linked.2 > > storing 1 entries at orphan.scan.erasure.linked.5 > > storing 1 entries at orphan.scan.erasure.linked.9 > > storing 1 entries at orphan.scan.erasure.linked.19 > > storing 1 entries at orphan.scan.erasure.linked.25 > > storing 1 entries at orphan.scan.erasure.linked.40 > > storing 1 entries at orphan.scan.erasure.linked.43 > > storing 1 entries at orphan.scan.erasure.linked.47 > > storing 1 entries at orphan.scan.erasure.linked.56 > > storing 1 entries at orphan.scan.erasure.linked.63 > > storing 1 entries at orphan.scan.erasure.linked.9 > > storing 1 entries at orphan.scan.erasure.linked.25 > > storing 1 entries at orphan.scan.erasure.linked.40 > > storing 1 entries at orphan.scan.erasure.linked.56 > > Thanks for your help > > -- > Yoann Moulin > EPFL IC-IT > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > -- Marius Vaitiekūnas
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
