Hi Vahideh,
I’m facing the same issue. Could you please share more details on your
solution? What do you mean by "changing the master zone" ?
Thanks!
Mahnoosh
On Sat, 25 Jan 2025, 05:48 Vahideh Alinouri, <vahideh.alino...@gmail.com>
wrote:
> The metadata sync issue has been resolved by changing the master zone and
> re-running the metadata sync.
>
> On Mon, Dec 23, 2024 at 2:15 PM Vahideh Alinouri <
> vahideh.alino...@gmail.com>
> wrote:
>
> > When I increased the debug level of the RGW sync client to 20, I get it:
> >
> > 2024-12-23T09:42:17.248+0000 7f124866b700 20 register_request
> > mgr=0x5633b4b8d958 req_data->id=79162099, curl_handle=0x5633bb89ee60
> > 2024-12-23T09:42:17.248+0000 7f124866b700 20 run: stack=0x5633b51952c0 is
> > io blocked
> > 2024-12-23T09:42:17.248+0000 7f1248e6c700 20 link_request
> > req_data=0x5633bc240d20 req_data->id=79162099, curl_handle=0x5633bb89ee60
> > 2024-12-23T09:42:17.252+0000 7f124866b700 20 rgw rados thread:
> > cr:s=0x5633b50f1540:op=0x5633b50bb100:25RGWMetaSyncShardControlCR:
> operate()
> > 2024-12-23T09:42:17.252+0000 7f124866b700 20 rgw rados thread:
> > cr:s=0x5633b50f1540:op=0x5633b520f400:18RGWMetaSyncShardCR: operate()
> > 2024-12-23T09:42:17.252+0000 7f124866b700 10 RGW-SYNC:meta:shard[0]:
> start
> > full sync
> > 2024-12-23T09:42:17.252+0000 7f124866b700 20 rgw rados thread:
> > cr:s=0x5633bb892000:op=0x5633b7731500:20RGWContinuousLeaseCR: operate()
> > 2024-12-23T09:42:17.252+0000 7f124866b700 20 rgw rados thread:
> > cr:s=0x5633b50f1540:op=0x5633b520f400:18RGWMetaSyncShardCR: operate()
> > 2024-12-23T09:42:17.252+0000 7f124866b700 20 run: stack=0x5633b50f1540
> > is_blocked_by_stack()=0 is_sleeping=1 waiting_for_child()=0
> > 2024-12-23T09:42:17.252+0000 7f124866b700 20 rgw rados thread:
> > cr:s=0x5633bb892000:op=0x5633b796ce00:20RGWSimpleRadosLockCR: operate()
> > 2024-12-23T09:42:17.252+0000 7f124866b700 20 rgw rados thread:
> > cr:s=0x5633bb892000:op=0x5633b796ce00:20RGWSimpleRadosLockCR: operate()
> > 2024-12-23T09:42:17.252+0000 7f124866b700 20 enqueued request
> > req=0x5633be14c500
> > 2024-12-23T09:42:17.252+0000 7f124866b700 20 RGWWQ:
> > 2024-12-23T09:42:17.252+0000 7f124866b700 20 req: 0x5633be14c500
> > 2024-12-23T09:42:17.252+0000 7f124866b700 20 run: stack=0x5633bb892000 is
> > io blocked
> > 2024-12-23T09:42:17.252+0000 7f125167d700 20 dequeued request
> > req=0x5633be14c500
> > 2024-12-23T09:42:17.252+0000 7f125167d700 20 RGWWQ: empty
> > 2024-12-23T09:42:17.252+0000 7f124866b700 20 rgw rados thread:
> > cr:s=0x5633bb892000:op=0x5633b796ce00:20RGWSimpleRadosLockCR: operate()
> > 2024-12-23T09:42:17.252+0000 7f124866b700 20 rgw rados thread:
> > cr:s=0x5633bb892000:op=0x5633b796ce00:20RGWSimpleRadosLockCR: operate()
> > 2024-12-23T09:42:17.252+0000 7f124866b700 20 rgw rados thread:
> > cr:s=0x5633bb892000:op=0x5633b796ce00:20RGWSimpleRadosLockCR: operate()
> > 2024-12-23T09:42:17.252+0000 7f124866b700 20 rgw rados thread:
> > cr:s=0x5633bb892000:op=0x5633b796ce00:20RGWSimpleRadosLockCR: operate()
> > 2024-12-23T09:42:17.252+0000 7f124866b700 20 rgw rados thread:
> > cr:s=0x5633bb892000:op=0x5633b7731500:20RGWContinuousLeaseCR: operate()
> > 2024-12-23T09:42:17.252+0000 7f124866b700 20 run: stack=0x5633bb892000 is
> > io blocked
> > 2024-12-23T09:42:17.252+0000 7f124866b700 20 rgw rados thread:
> > cr:s=0x5633b50f1540:op=0x5633b520f400:18RGWMetaSyncShardCR: operate()
> > 2024-12-23T09:42:17.252+0000 7f124866b700 10 RGW-SYNC:meta:shard[0]: took
> > lease
> > 2024-12-23T09:42:17.252+0000 7f124866b700 20 rgw rados thread:
> > cr:s=0x5633b50f1540:op=0x5633b796ce00:21RGWRadosGetOmapKeysCR: operate()
> > 2024-12-23T09:42:17.252+0000 7f124866b700 20 rgw rados thread:
> > cr:s=0x5633b50f1540:op=0x5633b796ce00:21RGWRadosGetOmapKeysCR: operate()
> > 2024-12-23T09:42:17.252+0000 7f124866b700 20 run: stack=0x5633b50f1540 is
> > io blocked
> > 2024-12-23T09:42:17.252+0000 7f124866b700 20 rgw rados thread:
> > cr:s=0x5633b50f1540:op=0x5633b796ce00:21RGWRadosGetOmapKeysCR: operate()
> > 2024-12-23T09:42:17.252+0000 7f124866b700 20 rgw rados thread:
> > cr:s=0x5633b50f1540:op=0x5633b796ce00:21RGWRadosGetOmapKeysCR: operate()
> > returned r=-2
> > 2024-12-23T09:42:17.252+0000 7f124866b700 20 rgw rados thread:
> > cr:s=0x5633b50f1540:op=0x5633b520f400:18RGWMetaSyncShardCR: operate()
> > 2024-12-23T09:42:17.252+0000 7f124866b700 0 meta sync: ERROR:
> > full_sync(): RGWRadosGetOmapKeysCR() returned ret=-2
> > 2024-12-23T09:42:17.252+0000 7f124866b700 0 RGW-SYNC:meta:shard[0]:
> > ERROR: failed to list omap keys, status=-2
> > 2024-12-23T09:42:17.252+0000 7f124866b700 20 rgw rados thread:
> > cr:s=0x5633b50f1540:op=0x5633b520f400:18RGWMetaSyncShardCR: operate()
> > 2024-12-23T09:42:17.252+0000 7f124866b700 20 run: stack=0x5633b50f1540
> > is_blocked_by_stack()=0 is_sleeping=0 waiting_for_child()=1
> > 2024-12-23T09:42:17.252+0000 7f124866b700 20 rgw rados thread:
> > cr:s=0x5633bb892000:op=0x5633b7731500:20RGWContinuousLeaseCR: operate()
> > 2024-12-23T09:42:17.252+0000 7f124866b700 20 rgw rados thread:
> > cr:s=0x5633bb892000:op=0x5633b796ce00:22RGWSimpleRadosUnlockCR: operate()
> > 2024-12-23T09:42:17.252+0000 7f124866b700 20 rgw rados thread:
> > cr:s=0x5633bb892000:op=0x5633b796ce00:22RGWSimpleRadosUnlockCR: operate()
> > 2024-12-23T09:42:17.252+0000 7f124866b700 20 enqueued request
> > req=0x5633bd10ea20
> >
> > And when I try to run this to get the list of OMAP, the output is empty!
> > rados -p s3-cdn-dc07.rgw.meta listomapvals -N root
> .bucket.meta.XXXX:XXXX.9
> >
> > When I list the PGs of meta pool using ceph pg ls-by-pool
> > s3-cdn-dc07.rgw.meta, There are some PGs that have BYTES size but
> > OMAP_BYTES and OMAP_KEYS values are 0!
> >
> > PG OBJECTS DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES*
> > OMAP_KEYS* LOG STATE SINCE VERSION REPORTED UP
> > ACTING SCRUB_STAMP DEEP_SCRUB_STAMP
> > LAST_SCRUB_DURATION SCRUB_SCHEDULING
> >
> > 10.0 14 0 0 0 3364 1676
> > 9 957 active+clean 6h 57938'32678 57939:9604093 [70,88,46]p70
> > [70,88,46]p70 2024-12-23T04:57:43.792352+0000
> > 2024-12-21T22:45:49.820084+0000 1 periodic scrub
> > scheduled @ 2024-12-24T16:29:19.844180+0000
> > *10.1 10 0 0 0 3073 0
> > 0 49 active+clean 14h 57715'25516 57939:8975816 [0,60,52]p0
> > [0,60,52]p0 2024-12-22T20:40:57.494355+0000
> > 2024-12-17T17:43:43.383320+0000 1 periodic scrub
> > scheduled @ 2024-12-23T22:30:45.498616+0000*
> > 10.2 15 0 0 0 3706 188
> > 1 132 active+clean 27h 57938'26080 57938:8966743 [39,14,1]p39
> > [39,14,1]p39 2024-12-22T08:05:34.147330+0000
> > 2024-12-22T08:05:34.147330+0000 1 periodic scrub
> > scheduled @ 2024-12-23T12:00:40.771526+0000
> > 10.3 14 0 0 0 3898 2234
> > 11 414 active+clean 2h 57939'36531 57939:9388345
> [40,78,6]p40
> > [40,78,6]p40 2024-12-23T09:00:29.100038+0000
> > 2024-12-16T12:58:31.054537+0000 1 periodic scrub
> > scheduled @ 2024-12-24T14:36:29.705903+0000
> >
> >
> >
> > *10.4 13 0 0 0 3346 0
> > 0 67 active+clean 25h 57715'25662 57939:8964482 [34,44,25]p34
> > [34,44,25]p34 2024-12-22T09:26:16.897803+0000
> > 2024-12-16T23:09:15.271909+0000 1 periodic scrub
> > scheduled @ 2024-12-23T20:57:45.809155+000010.5 13 0
> > 0 0 3567 0 0 2904 active+clean 28h
> > 57939'120020 57939:9016514 [66,37,91]p66 [66,37,91]p66
> > 2024-12-22T07:03:26.677870+0000 2024-12-19T09:16:25.987456+0000
> > 1 periodic scrub scheduled @
> 2024-12-23T17:12:34.661323+000010.6
> > 7 0 0 0 2159 0 0
> > 39 active+clean 15h 57938'25500 57939:9961651 [19,24,56]p19
> > [19,24,56]p19 2024-12-22T19:16:11.782123+0000
> > 2024-12-22T19:16:11.782123+0000 1 periodic scrub
> > scheduled @ 2024-12-24T01:40:30.212204+000010.7 20 0
> > 0 0 5963 0 0 116 active+clean 33m
> > 57715'25722 57939:9028884 [50,58,84]p50 [50,58,84]p50
> > 2024-12-23T10:35:45.237158+0000 2024-12-17T03:28:34.643774+0000
> > 1 periodic scrub scheduled @ 2024-12-24T18:39:23.977660+0000*
> > 10.8 20 0 0 0 6074 2155
> > 11 226 active+clean 9h 57938'25807 57939:9347768
> [22,4,42]p22
> > [22,4,42]p22 2024-12-23T02:07:21.060581+0000
> > 2024-12-21T23:16:53.596335+0000 1 periodic scrub
> > scheduled @ 2024-12-24T10:21:26.911522+0000
> > *10.9 10 0 0 0 3046 0
> > 0 63 active+clean 10h 57715'25597 57939:9021141 [10,38,15]p10
> > [10,38,15]p10 2024-12-23T00:19:34.157798+0000
> > 2024-12-20T11:28:20.294176+0000 1 periodic scrub
> > scheduled @ 2024-12-24T01:15:58.674425+0000*
> > 10.a 13 0 0 0 3400 188
> > 1 79 active+clean 6h 57715'25740 57938:9021299 [2,31,80]p2
> > [2,31,80]p2 2024-12-23T04:58:25.809790+0000
> > 2024-12-16T15:51:26.184478+0000 1 periodic scrub
> > scheduled @ 2024-12-24T05:19:15.221400+0000
> > 10.b 11 0 0 0 2915 176
> > 1 68 active+clean 3h 57854'25555 57939:9067491 [53,82,20]p53
> > [53,82,20]p53 2024-12-23T07:52:31.500216+0000
> > 2024-12-22T07:11:15.383987+0000 1 periodic scrub
> > scheduled @ 2024-12-24T11:42:16.438508+0000
> > 10.c 18 0 0 0 4430 2182
> > 11 458 active+clean 9h 57939'36171 57939:9036209
> [16,48,8]p16
> > [16,48,8]p16 2024-12-23T01:46:59.720703+0000
> > 2024-12-21T19:11:29.473994+0000 1 periodic scrub
> > scheduled @ 2024-12-24T11:00:21.726487+0000
> > 10.d 13 0 0 0 3135 187
> > 1 122 active+clean 8h 57901'26081 57939:9015503 [36,3,13]p36
> > [36,3,13]p36 2024-12-23T02:31:13.107381+0000
> > 2024-12-16T21:47:29.385720+0000 1 periodic scrub
> > scheduled @ 2024-12-24T10:01:35.665610+0000
> > *10.e 14 0 0 0 3366 0
> > 0 71 active+clean 7h 57715'25607 57939:9843618 [83,41,17]p83
> > [83,41,17]p83 2024-12-23T03:39:55.648905+0000
> > 2024-12-20T08:01:47.715401+0000 1 periodic scrub
> > scheduled @ 2024-12-24T09:58:14.031613+0000*
> > 10.f 14 0 0 0 4105 2508
> > 13 521 active+clean 20h 57939'39421 57939:8983895
> [33,49,68]p33
> > [33,49,68]p33 2024-12-22T14:39:32.975012+0000
> > 2024-12-22T14:39:32.975012+0000 1 periodic scrub
> > scheduled @ 2024-12-23T23:39:32.875651+0000
> >
> > I tried to fix them with ceph pg deep-scrub, but there were no changes!
> >
> > I started to check the primary OSD log of 10.1 in detail. I see the
> > following logs. Why
> >
> > 2024-12-23T00:03:38.920+0000 7f3fef2d7700 0 <cls>
> >
> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.2.5/rpm/el8/BUILD/ceph-17.2.5/src/cls/fifo/cls_fifo.cc:112:
> > ERROR: int
> > rados::cls::fifo::{anonymous}::read_part_header(cls_method_context_t,
> > rados::cls::fifo::part_header*): failed decoding part header
> > 2024-12-23T00:03:38.920+0000 7f3fef2d7700 0 <cls>
> >
> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.2.5/rpm/el8/BUILD/ceph-17.2.5/src/cls/fifo/cls_fifo.cc:781:
> > int rados::cls::fifo::{anonymous}::trim_part(cls_method_context_t,
> > ceph::buffer::v15_2_0::list*, ceph::buffer::v15_2_0::list*): failed to
> read
> > part header 2024-12-23T00:03:38.920+0000 7f3feb2cf700 0 <cls>
> >
> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.2.5/rpm/el8/BUILD/ceph-17.2.5/src/cls/fifo/cls_fifo.cc:112:
> > ERROR: int
> > rados::cls::fifo::{anonymous}::read_part_header(cls_method_context_t,
> > rados::cls::fifo::part_header*): failed decoding part header
> > 2024-12-23T00:03:38.920+0000 7f3feb2cf700 0 <cls>
> >
> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.2.5/rpm/el8/BUILD/ceph-17.2.5/src/cls/fifo/cls_fifo.cc:781:
> > int rados::cls::fifo::{anonymous}::trim_part(cls_method_context_t,
> > ceph::buffer::v15_2_0::list*, ceph::buffer::v15_2_0::list*): failed to
> read
> > part header 2024-12-23T00:03:38.920+0000 7f3feb2cf700 0 <cls>
> >
> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.2.5/rpm/el8/BUILD/ceph-17.2.5/src/cls/fifo/cls_fifo.cc:112:
> > ERROR: int
> > rados::cls::fifo::{anonymous}::read_part_header(cls_method_context_t,
> > rados::cls::fifo::part_header*): failed decoding part header
> > 2024-12-23T00:03:38.920+0000 7f3feb2cf700 0 <cls>
> >
> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.2.5/rpm/el8/BUILD/ceph-17.2.5/src/cls/fifo/cls_fifo.cc:781:
> > int rados::cls::fifo::{anonymous}::trim_part(cls_method_context_t,
> > ceph::buffer::v15_2_0::list*, ceph::buffer::v15_2_0::list*): failed to
> read
> > part header
> >
> > On Fri, Dec 20, 2024 at 10:59 AM Eugen Block <ebl...@nde.ag> wrote:
> >
> >> I don't really have a good idea, except for maybe running "metadata
> >> sync init" and "metadata sync run"?
> >> But I wanted to point out that the .meta pool uses namespaces.
> >>
> >> > - Attempted to list metadata in the pool using rados ls -p
> >> > s3-cdn-dc07.rgw.meta, but got an empty result.
> >>
> >> Try this instead:
> >>
> >> rados -p s3-cdn-dc07.rgw.meta ls --all
> >>
> >> Do you have a specific object where the omap data is missing? Maybe
> >> increase debug_rgw logs to find a trace why it's missing.
> >>
> >> Zitat von Vahideh Alinouri <vahideh.alino...@gmail.com>:
> >>
> >> > I also see this in the output of radosgw-admin metadata sync status. I
> >> > think it's strange because there should be a marker to follow the
> sync.
> >> > {
> >> > "key": 0,
> >> > "val": {
> >> > "state": 0,
> >> > "marker": "",
> >> > "next_step_marker":
> >> "1_1730469205.875723_877487777.1",
> >> > "total_entries": 174,
> >> > "pos": 0,
> >> > "timestamp": "2024-11-01T13:53:25.875723Z",
> >> > "realm_epoch": 0
> >> > }
> >> >
> >> > On Mon, Dec 16, 2024 at 1:24 PM Vahideh Alinouri <
> >> vahideh.alino...@gmail.com>
> >> > wrote:
> >> >
> >> >> I also see this log in the RGW log:
> >> >>
> >> >> 2024-12-16T12:23:58.651+0000 7f9b2b9fe700 1 ====== starting new
> >> request
> >> >> req=0x7f9ad9959730 =====
> >> >> 2024-12-16T12:23:58.651+0000 7f9b2b9fe700 -1 req 11778501317150336521
> >> >> 0.000000000s :list_data_changes_log int
> >> >> rgw::cls::fifo::{anonymous}::list_part(const DoutPrefixProvider*,
> >> >> librados::v14_2_0::IoCtx&, const string&,
> >> >> std::optional<std::basic_string_view<char> >, uint64_t, uint64_t,
> >> >> std::vector<rados::cls::fifo::part_list_entry>*, bool*, bool*,
> >> >> std::string*, uint64_t, optional_yield):245 fifo::op::LIST_PART
> failed
> >> >> r=-34 tid=4176
> >> >> 2024-12-16T12:23:58.651+0000 7f9b2b9fe700 -1 req 11778501317150336521
> >> >> 0.000000000s :list_data_changes_log int
> >> rgw::cls::fifo::FIFO::list(const
> >> >> DoutPrefixProvider*, int, std::optional<std::basic_string_view<char>
> >,
> >> >> std::vector<rgw::cls::fifo::list_entry>*, bool*, optional_yield):1660
> >> >> list_entries failed: r=-34 tid= 4176
> >> >> 2024-12-16T12:23:58.651+0000 7f9b2b9fe700 -1 req 11778501317150336521
> >> >> 0.000000000s :list_data_changes_log virtual int
> >> >> RGWDataChangesFIFO::list(const DoutPrefixProvider*, int, int,
> >> >> std::vector<rgw_data_change_log_entry>&,
> >> >> std::optional<std::basic_string_view<char> >, std::string*, bool*):
> >> unable
> >> >> to list FIFO: data_log.44: (34) Numerical result out of range
> >> >>
> >> >> On Sun, Dec 15, 2024 at 10:45 PM Vahideh Alinouri <
> >> >> vahideh.alino...@gmail.com> wrote:
> >> >>
> >> >>> Hi guys,
> >> >>>
> >> >>> My Ceph release is Quincy 17.2.5. I need to change the master zone
> to
> >> >>> decommission the old one and upgrade all zones. I have separated the
> >> client
> >> >>> traffic and sync traffic in RGWs, meaning there are separate RGW
> >> daemons
> >> >>> handling the sync process.
> >> >>>
> >> >>> I encountered an issue when trying to sync one of the zones in the
> >> >>> zonegroup. The data sync is proceeding fine, but I have an issue
> with
> >> the
> >> >>> metadata sync. It gets stuck behind on a shard. Here is the output
> >> >>> from radosgw-admin
> >> >>> sync status:
> >> >>>
> >> >>> metadata sync syncing
> >> >>> full sync: 1/64 shards
> >> >>> full sync: 135 entries to sync
> >> >>> incremental sync: 63/64 shards
> >> >>> metadata is behind on 1 shard
> >> >>> behind shards: [0]
> >> >>>
> >> >>> In the RGW log, I see this error:
> >> >>> 2024-12-15T21:30:59.641+0000 7f6dff472700 1 beast: 0x7f6d2f1cf730:
> >> >>> 172.19.66.112 - s3-cdn-user [15/Dec/2024:21:30:59.641 +0000] "GET
> >> >>>
> >>
> /admin/log/?type=data&id=56&marker=00000000000000000000%3A00000000000000204086&extra-info=true&rgwx-zonegroup=7c01d60f-88c6-4192-baf7-d725260bf05d
> >> >>> HTTP/1.1" 200 44 - - - latency=0.000000000s
> >> >>> 2024-12-15T21:30:59.701+0000 7f6e44d1e700 0 meta sync: ERROR:
> >> >>> full_sync(): RGWRadosGetOmapKeysCR() returned ret=-2
> >> >>> 2024-12-15T21:30:59.701+0000 7f6e44d1e700 0 RGW-SYNC:meta:shard[0]:
> >> >>> ERROR: failed to list omap keys, status=-2
> >> >>> 2024-12-15T21:30:59.701+0000 7f6e44d1e700 0 meta sync: ERROR:
> >> >>> RGWBackoffControlCR called coroutine returned -2
> >> >>> 2024-12-15T21:31:00.705+0000 7f6e44d1e700 0 meta sync: ERROR:
> >> >>> full_sync(): RGWRadosGetOmapKeysCR() returned ret=-2
> >> >>> 2024-12-15T21:31:00.705+0000 7f6e44d1e700 0 RGW-SYNC:meta:shard[0]:
> >> >>> ERROR: failed to list omap keys, status=-2
> >> >>> 2024-12-15T21:31:00.705+0000 7f6e44d1e700 0 meta sync: ERROR:
> >> >>> RGWBackoffControlCR called coroutine returned -2
> >> >>>
> >> >>> I’ve tried the following steps:
> >> >>>
> >> >>> - Changed the PG number of the metadata pool to force a
> rebalance,
> >> >>> but everything was fine.
> >> >>> - Ran metadata sync init and tried to run it again.
> >> >>> - Restarted RGW services in both the zone and the master zone.
> >> >>> - Created a user in the master zone to ensure metadata sync
> works,
> >> >>> which was successful.
> >> >>> - Checked OSD logs but didn’t see any specific errors.
> >> >>> - Attempted to list metadata in the pool using rados ls -p
> >> >>> s3-cdn-dc07.rgw.meta, but got an empty result.
> >> >>> - Compared the code for listing OMAP keys between Quincy and
> Squid
> >> >>> versions; there were no specific changes.
> >> >>>
> >> >>> I’m looking for any advice or suggestions to resolve this issue.
> >> >>>
> >> >>>
> >> > _______________________________________________
> >> > ceph-users mailing list -- ceph-users@ceph.io
> >> > To unsubscribe send an email to ceph-users-le...@ceph.io
> >>
> >>
> >> _______________________________________________
> >> ceph-users mailing list -- ceph-users@ceph.io
> >> To unsubscribe send an email to ceph-users-le...@ceph.io
> >>
> >
> _______________________________________________
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
