The deep scrub of the pg updated the cluster that the large omap was gone. HEALTH_OK !
On Fri., Sep. 20, 2019, 2:31 p.m. shubjero, <shubj...@gmail.com> wrote: > Still trying to solve this one. > > Here is the corresponding log entry when the large omap object was found: > > ceph-osd.1284.log.2.gz:2019-09-18 11:43:39.237 7fcd68f96700 0 > log_channel(cluster) log [WRN] : Large omap object found. Object: > 26:86e4c833:::usage.22:head Key count: 2009548 Size (bytes): 369641376 > > I have since trimmed the entire usage log and disabled it entirely. > You can see from the output below that there's nothing in these usage > log objects. > > for i in `rados -p .usage ls`; do echo $i; rados -p .usage > listomapkeys $i | wc -l; done > usage.29 > 0 > usage.12 > 0 > usage.1 > 0 > usage.26 > 0 > usage.20 > 0 > usage.24 > 0 > usage.16 > 0 > usage.15 > 0 > usage.3 > 0 > usage.19 > 0 > usage.23 > 0 > usage.5 > 0 > usage.11 > 0 > usage.7 > 0 > usage.30 > 0 > usage.18 > 0 > usage.21 > 0 > usage.27 > 0 > usage.13 > 0 > usage.22 > 0 > usage.25 > 0 > . > 4 > usage.10 > 0 > usage.8 > 0 > usage.9 > 0 > usage.28 > 0 > usage.2 > 0 > usage.4 > 0 > usage.6 > 0 > usage.31 > 0 > usage.17 > 0 > > > root@infra:~# rados -p .usage listomapkeys usage.22 > root@infra:~# > > > On Thu, Sep 19, 2019 at 12:54 PM Charles Alva <charlesa...@gmail.com> > wrote: > > > > Could you please share how you trimmed the usage log? > > > > Kind regards, > > > > Charles Alva > > Sent from Gmail Mobile > > > > > > On Thu, Sep 19, 2019 at 11:46 PM shubjero <shubj...@gmail.com> wrote: > >> > >> Hey all, > >> > >> Yesterday our cluster went in to HEALTH_WARN due to 1 large omap > >> object in the .usage pool (I've posted about this in the past). Last > >> time we resolved the issue by trimming the usage log below the alert > >> threshold but this time it seems like the alert wont clear even after > >> trimming and (this time) disabling the usage log entirely. > >> > >> ceph health detail > >> HEALTH_WARN 1 large omap objects > >> LARGE_OMAP_OBJECTS 1 large omap objects > >> 1 large objects found in pool '.usage' > >> Search the cluster log for 'Large omap object found' for more > details. > >> > >> I've bounced ceph-mon, ceph-mgr, radosgw and even issued osd scrub on > >> the two osd's that hold pg's for the .usage pool but the alert wont > >> clear. > >> > >> It's been over 24 hours since I trimmed the usage log. > >> > >> Any suggestions? > >> > >> Jared Baker > >> Cloud Architect, OICR > >> _______________________________________________ > >> ceph-users mailing list -- ceph-users@ceph.io > >> To unsubscribe send an email to ceph-users-le...@ceph.io >
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
