On Wed, Jul 10, 2024 at 6:23 PM Richard Bade <hitr...@gmail.com> wrote:
>
> Hi Casey,
> Thanks for that info on the bilog. I'm in a similar situation with
> large omap objects and we have also had to reshard buckets on
> multisite losing the index on the secondary.
> We also now have a lot of buckets with sync disable so I wanted to
> check that it's always safe to trim the bilog on buckets with sync
> disabled?
in general, the only risk of trimming bilogs is that they refer to
changes you still want to replicate
after `bucket sync disable`, it's fine to trim if it isn't happening
automatically. it's fine even if you want to `bucket sync enable`
later, because that restarts a 'full sync' and skips any bilog entries
from before
> I can see some stale entries with "completed" state and a timestamp of
> a number of months ago but also some that say pending and have no
> timestamp.
>
> Istvan, I can also possibly help with your orphaned 40TB on the secondary
> zone.
> Each object has the bucket marker in its name. If you do a `rados -p
> {pool_name} ls` and find all the ones that start with the bucket
> marker (found with `radosgw-admin bucket stats
> --bucket={bucket_name}`) then you can do one of two things:
> 1, `rados rm` the object
> 2, restore the index with info from the object itself
> - create a dummy index template (use `radosgw-admin bi get` on a
> known good index to get the structure)
> - grab the etag from the object xattribs and use this and the name
> in the template (`rados -p {pool} getxattr {objname} user.rgw.etag`)
> - use ` radosgw-admin bi put` to create the index
> - use `radosgw-admin bucket check --check-objects --fix
> --bucket={bucket_name}` to fix up the bucket object count and object
> sizes at the end
>
> This process takes quite some time and I can't say if it's 100%
> perfect but it enabled us to get to a state where we could delete the
> buckets and clean up the objects.
> I hope this helps.
>
> Regards,
> Richard
>
> On Thu, 11 Jul 2024 at 01:25, Casey Bodley <cbod...@redhat.com> wrote:
> >
> > On Tue, Jul 9, 2024 at 12:41 PM Szabo, Istvan (Agoda)
> > <istvan.sz...@agoda.com> wrote:
> > >
> > > Hi Casey,
> > >
> > > 1.
> > > Regarding versioning, the user doesn't use verisoning it if I'm not
> > > mistaken:
> > > https://gist.githubusercontent.com/Badb0yBadb0y/d80c1bdb8609088970413969826d2b7d/raw/baee46865178fff454c224040525b55b54e27218/gistfile1.txt
> > >
> > > 2.
> > > Regarding multiparts, if it would have multipart thrash, it would be
> > > listed here:
> > > https://gist.githubusercontent.com/Badb0yBadb0y/d80c1bdb8609088970413969826d2b7d/raw/baee46865178fff454c224040525b55b54e27218/gistfile1.txt
> > > as a rgw.multimeta under the usage, right?
> > >
> > > 3.
> > > Regarding the multisite idea, this bucket has been a multisite bucket
> > > last year but we had to reshard (accepting to loose the replica on the
> > > 2nd site and just keep it in the master site) and that time as expected
> > > it has disappeared completely from the 2nd site (I guess the 40TB thrash
> > > still there but can't really find it how to clean 🙁 ). Now it is a single
> > > site bucket.
> > > Also it is the index pool, multisite logs should go to the rgw.log pool
> > > shouldn't it?
> >
> > some replication logs are in the log pool, but the per-object logs are
> > stored in the bucket index objects. you can inspect these with
> > `radosgw-admin bilog list --bucket=X`. by default, that will only list
> > --max-entries=1000. you can add --shard-id=Y to look at specific
> > 'large omap' objects
> >
> > even if your single-site bucket doesn't exist on the secondary zone,
> > changes on the primary zone are probably still generating these bilog
> > entries. you would need to do something like `radosgw-admin bucket
> > sync disable --bucket=X` to make it stop. because you don't expect
> > these changes to replicate, it's safe to delete any of this bucket's
> > bilog entries with `radosgw-admin bilog trim --end-marker 9
> > --bucket=X`. depending on ceph version, you may need to run this trim
> > command in a loop until the `bilog list` output is empty
> >
> > radosgw does eventually trim bilogs in the background after they're
> > processed, but the secondary zone isn't processing them in this case
> >
> > >
> > > Thank you
> > >
> > >
> > > ________________________________
> > > From: Casey Bodley <cbod...@redhat.com>
> > > Sent: Tuesday, July 9, 2024 10:39 PM
> > > To: Szabo, Istvan (Agoda) <istvan.sz...@agoda.com>
> > > Cc: Eugen Block <ebl...@nde.ag>; ceph-users@ceph.io <ceph-users@ceph.io>
> > > Subject: Re: [ceph-users] Re: Large omap in index pool even if properly
> > > sharded and not "OVER"
> > >
> > > Email received from the internet. If in doubt, don't click any link nor
> > > open any attachment !
> > > ________________________________
> > >
> > > in general, these omap entries should be evenly spread over the
> > > bucket's index shard objects. but there are two features that may
> > > cause entries to clump on a single shard:
> > >
> > > 1. for versioned buckets, multiple versions of the same object name
> > > map to the same index shard. this can become an issue if an
> > > application is repeatedly overwriting an object without cleaning up
> > > old versions. lifecycle rules can help to manage these noncurrent
> > > versions
> > >
> > > 2. during a multipart upload, all of the parts are tracked on the same
> > > index shard as the final object name. if applications are leaving a
> > > lot of incomplete multipart uploads behind (especially if they target
> > > the same object name) this can lead to similar clumping. the S3 api
> > > has operations to list and abort incomplete multipart uploads, along
> > > with lifecycle rules to automate their cleanup
> > >
> > > separately, multisite clusters use these same index shards to store
> > > replication logs. if sync gets far enough behind, these log entries
> > > can also lead to large omap warnings
> > >
> > > On Tue, Jul 9, 2024 at 10:25 AM Szabo, Istvan (Agoda)
> > > <istvan.sz...@agoda.com> wrote:
> > > >
> > > > It's the same bucket:
> > > > https://gist.github.com/Badb0yBadb0y/d80c1bdb8609088970413969826d2b7d
> > > >
> > > >
> > > > ________________________________
> > > > From: Eugen Block <ebl...@nde.ag>
> > > > Sent: Tuesday, July 9, 2024 8:03 PM
> > > > To: Szabo, Istvan (Agoda) <istvan.sz...@agoda.com>
> > > > Cc: ceph-users@ceph.io <ceph-users@ceph.io>
> > > > Subject: Re: [ceph-users] Re: Large omap in index pool even if properly
> > > > sharded and not "OVER"
> > > >
> > > > Email received from the internet. If in doubt, don't click any link nor
> > > > open any attachment !
> > > > ________________________________
> > > >
> > > > Are those three different buckets? Could you share the stats for each
> > > > of them?
> > > >
> > > > radosgw-admin bucket stats --bucket=<BUCKET>
> > > >
> > > > Zitat von "Szabo, Istvan (Agoda)" <istvan.sz...@agoda.com>:
> > > >
> > > > > Hello,
> > > > >
> > > > > Yeah, still:
> > > > >
> > > > > the .dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.151 | wc -l
> > > > > 290005
> > > > >
> > > > > and the
> > > > > .dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.726 | wc -l
> > > > > 289378
> > > > >
> > > > > And just make me happy more I have one more
> > > > > .dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.6 | wc -l
> > > > > 181588
> > > > >
> > > > > This is my crush tree (I'm using host based crush rule)
> > > > > https://gist.githubusercontent.com/Badb0yBadb0y/9bea911701184a51575619bc99cca94d/raw/e5e4a918d327769bb874aaed279a8428fd7150d5/gistfile1.txt
> > > > >
> > > > > I'm thinking could that be the issue that host 2s13-15 has less nvme
> > > > > osd (however size wise same as in the other 12 host where have 8x
> > > > > nvme osd) than the others?
> > > > > But the pgs are located like this:
> > > > >
> > > > > pg26.427
> > > > > osd.261 host8
> > > > > osd.488 host13
> > > > > osd.276 host4
> > > > >
> > > > > pg26.606
> > > > > osd.443 host12
> > > > > osd.197 host8
> > > > > osd.524 host14
> > > > >
> > > > > pg26.78c
> > > > > osd.89 host7
> > > > > osd.406 host11
> > > > > osd.254 host6
> > > > >
> > > > > If pg26.78c wouldn't be here I'd say 100% the nvme osd distribution
> > > > > based on host is the issue, however this pg is not located on any of
> > > > > the 4x nvme osd nodes 😕
> > > > >
> > > > > Ty
> > > > >
> > > > > ________________________________
> > > > > From: Eugen Block <ebl...@nde.ag>
> > > > > Sent: Tuesday, July 9, 2024 6:02 PM
> > > > > To: ceph-users@ceph.io <ceph-users@ceph.io>
> > > > > Subject: [ceph-users] Re: Large omap in index pool even if properly
> > > > > sharded and not "OVER"
> > > > >
> > > > > Email received from the internet. If in doubt, don't click any link
> > > > > nor open any attachment !
> > > > > ________________________________
> > > > >
> > > > > Hi,
> > > > >
> > > > > the number of shards looks fine, maybe this was just a temporary
> > > > > burst? Did you check if the rados objects in the index pool still have
> > > > > more than 200k omap objects? I would try someting like
> > > > >
> > > > > rados -p <index_pool> listomapkeys
> > > > > .dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.151 | wc -l
> > > > >
> > > > >
> > > > > Zitat von "Szabo, Istvan (Agoda)" <istvan.sz...@agoda.com>:
> > > > >
> > > > >> Hi,
> > > > >>
> > > > >> I have a pretty big bucket which sharded with 1999 shard so in
> > > > >> theory can hold close to 200m objects (199.900.000).
> > > > >> Currently it has 54m objects.
> > > > >>
> > > > >> Bucket limit check looks also good:
> > > > >> "bucket": ""xyz,
> > > > >> "tenant": "",
> > > > >> "num_objects": 53619489,
> > > > >> "num_shards": 1999,
> > > > >> "objects_per_shard": 26823,
> > > > >> "fill_status": "OK"
> > > > >>
> > > > >> This is the bucket id:
> > > > >> "id": "9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1"
> > > > >>
> > > > >> This is the log lines:
> > > > >> 2024-06-27T10:41:05.679870+0700 osd.261 (osd.261) 9643 : cluster
> > > > >> [WRN] Large omap object found. Object:
> > > > >> 26:e433e65c:::.dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.151:head
> > > > >> PG: 26.3a67cc27 (26.427) Key count: 236919 Size
> > > > >> (bytes):
> > > > >> 89969920
> > > > >>
> > > > >> 2024-06-27T10:43:35.557835+0700 osd.89 (osd.89) 9000 : cluster [WRN]
> > > > >> Large omap object found. Object:
> > > > >> 26:31ff4df1:::.dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.726:head
> > > > >> PG: 26.8fb2ff8c (26.78c) Key count: 236495 Size
> > > > >> (bytes):
> > > > >> 95560458
> > > > >>
> > > > >> Tried to deep scrub the affected pgs, tried to deep-scrub the
> > > > >> mentioned osds in the log but didn't help.
> > > > >> Why? What I'm missing?
> > > > >>
> > > > >> Thank you in advance for your help.
> > > > >>
> > > > >> ________________________________
> > > > >> This message is confidential and is for the sole use of the intended
> > > > >> recipient(s). It may also be privileged or otherwise protected by
> > > > >> copyright or other legal rules. If you have received it by mistake
> > > > >> please let us know by reply email and delete it from your system. It
> > > > >> is prohibited to copy this message or disclose its content to
> > > > >> anyone. Any confidentiality or privilege is not waived or lost by
> > > > >> any mistaken delivery or unauthorized disclosure of the message. All
> > > > >> messages sent to and from Agoda may be monitored to ensure
> > > > >> compliance with company policies, to protect the company's interests
> > > > >> and to remove potential malware. Electronic messages may be
> > > > >> intercepted, amended, lost or deleted, or contain viruses.
> > > > >> _______________________________________________
> > > > >> ceph-users mailing list -- ceph-users@ceph.io
> > > > >> To unsubscribe send an email to ceph-users-le...@ceph.io
> > > > >
> > > > >
> > > > > _______________________________________________
> > > > > ceph-users mailing list -- ceph-users@ceph.io
> > > > > To unsubscribe send an email to ceph-users-le...@ceph.io
> > > > >
> > > > > ________________________________
> > > > > This message is confidential and is for the sole use of the intended
> > > > > recipient(s). It may also be privileged or otherwise protected by
> > > > > copyright or other legal rules. If you have received it by mistake
> > > > > please let us know by reply email and delete it from your system. It
> > > > > is prohibited to copy this message or disclose its content to
> > > > > anyone. Any confidentiality or privilege is not waived or lost by
> > > > > any mistaken delivery or unauthorized disclosure of the message. All
> > > > > messages sent to and from Agoda may be monitored to ensure
> > > > > compliance with company policies, to protect the company's interests
> > > > > and to remove potential malware. Electronic messages may be
> > > > > intercepted, amended, lost or deleted, or contain viruses.
> > > >
> > > >
> > > >
> > > >
> > > > ________________________________
> > > > This message is confidential and is for the sole use of the intended
> > > > recipient(s). It may also be privileged or otherwise protected by
> > > > copyright or other legal rules. If you have received it by mistake
> > > > please let us know by reply email and delete it from your system. It is
> > > > prohibited to copy this message or disclose its content to anyone. Any
> > > > confidentiality or privilege is not waived or lost by any mistaken
> > > > delivery or unauthorized disclosure of the message. All messages sent
> > > > to and from Agoda may be monitored to ensure compliance with company
> > > > policies, to protect the company's interests and to remove potential
> > > > malware. Electronic messages may be intercepted, amended, lost or
> > > > deleted, or contain viruses.
> > > > _______________________________________________
> > > > ceph-users mailing list -- ceph-users@ceph.io
> > > > To unsubscribe send an email to ceph-users-le...@ceph.io
> > >
> > _______________________________________________
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
>
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
