Dear All,
My ceph cluster is giving Large OMAP warning from approx 2-3 Months. I
tried a few things like :
*Deep scrub of PGs*
*Compact OSDs*
*Trim log*
But these didn't work out.
I guess the main issue is that 4 shards in replication site are always
recovering from 2-3 months.
Any suggestions are highly appreciated.
Sync status:
root@drhost1:~# radosgw-admin sync status
realm e259e0a92 (object-storage)
zonegroup 7a8606d2 (staas)
zone c8022ad1 (repstaas)
metadata sync syncing
full sync: 0/64 shards
incremental sync: 64/64 shards
metadata is caught up with master
data sync source: 2072ee2d32 (masterstaas)
syncing
full sync: 0/128 shards
incremental sync: 128/128 shards
data is behind on 3 shards
behind shards: [7,90,100]
oldest incremental change not applied:
2025-03-12T13:14:10.268469+0530 [7]
4 shards are recovering
recovering shards: [31,41,55,80]
Master site:
1. *root@master1:~# for obj in $(rados ls -p masterstaas.rgw.log); do echo
"$(rados listomapkeys -p masterstaas.rgw.log $obj | wc -l) $obj";done |
sort -nr | head -10*
1225387 data_log.91
1225065 data_log.86
1224662 data_log.87
1224448 data_log.92
1224018 data_log.89
1222156 data_log.93
1201489 data_log.83
1174125 data_log.90
363498 data_log.84
258709 data_log.6
2. *root@master1:~# for obj in data_log.91 data_log.86 data_log.87
data_log.92 data_log.89 data_log.93 data_log.83 data_log.90; do rados stat
-p masterstaas.rgw.log $obj; done*
masterstaas.rgw.log/data_log.91 mtime 2025-02-24T15:09:25.000000+0530, size
0
masterstaas.rgw.log/data_log.86 mtime 2025-02-24T15:01:25.000000+0530, size
0
masterstaas.rgw.log/data_log.87 mtime 2025-02-24T15:02:25.000000+0530, size
0
masterstaas.rgw.log/data_log.92 mtime 2025-02-24T15:11:01.000000+0530, size
0
masterstaas.rgw.log/data_log.89 mtime 2025-02-24T14:54:55.000000+0530, size
0
masterstaas.rgw.log/data_log.93 mtime 2025-02-24T14:53:25.000000+0530, size
0
masterstaas.rgw.log/data_log.83 mtime 2025-02-24T14:16:21.000000+0530, size
0
masterstaas.rgw.log/data_log.90 mtime 2025-02-24T15:05:25.000000+0530, size
0
*3. ceph cluster log :*
2025-02-22T04:18:27.324886+0530 osd.173 (osd.173) 19 : cluster [WRN] Large
omap object found. Object: 124:b2ddf551:::data_log.93:head PG: 124.8aafbb4d
(124.d) Key count: 1218170 Size (bytes): 297085860
2025-02-22T04:18:28.735886+0530 osd.65 (osd.65) 308 : cluster [WRN] Large
omap object found. Object: 124:f2081d70:::data_log.92:head PG: 124.eb8104f
(124.f) Key count: 1220420 Size (bytes): 295240028
2025-02-22T04:18:30.668884+0530 mon.master1 (mon.0) 7974038 : cluster [WRN]
Health check update: 3 large omap objects (LARGE_OMAP_OBJECTS)
2025-02-22T04:18:31.127585+0530 osd.18 (osd.18) 224 : cluster [WRN] Large
omap object found. Object: 124:d1061236:::data_log.86:head PG: 124.6c48608b
(124.b) Key count: 1221047 Size (bytes): 295398274
2025-02-22T04:18:33.189561+0530 osd.37 (osd.37) 32665 : cluster [WRN] Large
omap object found. Object: 124:9a2e04b7:::data_log.87:head PG: 124.ed207459
(124.19) Key count: 1220584 Size (bytes): 295290366
2025-02-22T04:18:35.007117+0530 osd.77 (osd.77) 135 : cluster [WRN] Large
omap object found. Object: 124:6b9e929a:::data_log.89:head PG: 124.594979d6
(124.16) Key count: 1219913 Size (bytes): 295127816
2025-02-22T04:18:36.189141+0530 mon.master1 (mon.0) 7974039 : cluster [WRN]
Health check update: 5 large omap objects (LARGE_OMAP_OBJECTS)
2025-02-22T04:18:36.340247+0530 osd.112 (osd.112) 259 : cluster [WRN] Large
omap object found. Object: 124:0958bece:::data_log.83:head PG: 124.737d1a90
(124.10) Key count: 1200406 Size (bytes): 290280292
2025-02-22T04:18:38.523766+0530 osd.73 (osd.73) 1064 : cluster [WRN] Large
omap object found. Object: 124:fddd971f:::data_log.91:head PG: 124.f8e9bbbf
(124.3f) Key count: 1221183 Size (bytes): 295425320
2025-02-22T04:18:42.619926+0530 osd.92 (osd.92) 285 : cluster [WRN] Large
omap object found. Object: 124:7dc404fa:::data_log.90:head PG: 124.5f2023be
(124.3e) Key count: 1169895 Size (bytes): 283025576
2025-02-22T04:18:44.242655+0530 mon.master1 (mon.0) 7974043 : cluster [WRN]
Health check update: 8 large omap objects (LARGE_OMAP_OBJECTS)
Replica site:
1. *for obj in $(rados ls -p repstaas.rgw.log); do echo "$(rados
listomapkeys -p repstaas.rgw.log $obj | wc -l) $obj";done | sort -nr | head
-10*
432850 data_log.91
432384 data_log.87
432323 data_log.93
431783 data_log.86
431510 data_log.92
427959 data_log.89
414522 data_log.90
407571 data_log.83
151015 data_log.84
109790 data_log.4
2. *ceph cluster log:*
grep -ir "Large omap object found" /var/log/ceph/
/var/log/ceph/ceph-mon.drhost1.log:2025-03-12T14:49:59.997+0530
7fc4ad544700 0 log_channel(cluster) log [WRN] : Search the cluster log
for 'Large omap object found' for more details.
/var/log/ceph/ceph.log:2025-03-12T14:49:02.078108+0530 osd.10 (osd.10) 21 :
cluster [WRN] Large omap object found. Object:
6:b2ddf551:::data_log.93:head PG: 6.8aafbb4d (6.d) Key count: 432323 Size
(bytes): 105505884
/var/log/ceph/ceph.log:2025-03-12T14:49:02.389288+0530 osd.48 (osd.48) 37 :
cluster [WRN] Large omap object found. Object:
6:d1061236:::data_log.86:head PG: 6.6c48608b (6.b) Key count: 431782 Size
(bytes): 104564674
/var/log/ceph/ceph.log:2025-03-12T14:49:07.166954+0530 osd.24 (osd.24) 24 :
cluster [WRN] Large omap object found. Object:
6:0958bece:::data_log.83:head PG: 6.737d1a90 (6.10) Key count: 407571 Size
(bytes): 98635522
/var/log/ceph/ceph.log:2025-03-12T14:49:09.100110+0530 osd.63 (osd.63) 5 :
cluster [WRN] Large omap object found. Object:
6:9a2e04b7:::data_log.87:head PG: 6.ed207459 (6.19) Key count: 432384 Size
(bytes): 104712350
/var/log/ceph/ceph.log:2025-03-12T14:49:08.703760+0530 osd.59 (osd.59) 11 :
cluster [WRN] Large omap object found. Object:
6:6b9e929a:::data_log.89:head PG: 6.594979d6 (6.16) Key count: 427959 Size
(bytes): 103773777
/var/log/ceph/ceph.log:2025-03-12T14:49:11.126132+0530 osd.40 (osd.40) 24 :
cluster [WRN] Large omap object found. Object:
6:f2081d70:::data_log.92:head PG: 6.eb8104f (6.f) Key count: 431508 Size
(bytes): 104520406
/var/log/ceph/ceph.log:2025-03-12T14:49:13.799473+0530 osd.43 (osd.43) 61 :
cluster [WRN] Large omap object found. Object:
6:fddd971f:::data_log.91:head PG: 6.f8e9bbbf (6.1f) Key count: 432850 Size
(bytes): 104418869
/var/log/ceph/ceph.log:2025-03-12T14:49:14.398480+0530 osd.3 (osd.3) 55 :
cluster [WRN] Large omap object found. Object:
6:7dc404fa:::data_log.90:head PG: 6.5f2023be (6.1e) Key count: 414521 Size
(bytes): 100396561
/var/log/ceph/ceph.log:2025-03-12T14:50:00.000484+0530 mon.drhost1 (mon.0)
207423 : cluster [WRN] Search the cluster log for 'Large omap object
found' for more details.
Regards,
Danish
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
