Hi Mark,
Thanks for your response. I did manual compaction on all osds using
ceph-kvstore-tool. It reduced the number of slow ops but It didn't solve
the problem completely.
On Mon, Jul 26, 2021 at 8:06 PM Mark Nelson wrote:
> Yeah, I suspect that regular manual compaction might be the necessary
Yeah, I suspect that regular manual compaction might be the necessary
work around here if tombstones are slowing down iterator performance.
If it is related to tombstones, it would be similar to what we saw when
we tried to use deleterange and saw similar performance issues.
I'm a little at a
Hi Mahnoosh!
Unfortunately I'm not an expert in RGW hence nothing to recommend from
this side.
Apparently your issues are caused by bulk data removal - it appears that
RocksDB can hardly sustain such things and its performance degrades.
We've seen that plenty of times before.
So far there
Thanks for your help. Our hdd osd have separate nvme disks for DB use.
On Mon, Jul 26, 2021 at 3:49 PM Igor Fedotov wrote:
> Unfortunately I'm not an expert in RGW hence nothing to recommend from
> that side.
>
> Apparently your issues are caused by bulk data removal - it appears that
> RocksDB
Unfortunately I'm not an expert in RGW hence nothing to recommend from
that side.
Apparently your issues are caused by bulk data removal - it appears that
RocksDB can hardly sustain such things and its performance degrades.
We've seen that plenty of times before.
So far there are two known w
Hi Igor,
Thanks for your response.This problem happens on my osds with hdd disks. I
set the bluefs_buffered_io to true just for these osds but it caused my
bucket index disks (which are ssd) to produce slow ops. I also tried to set
bluefs_buffered_io to true in bucket index osds but they filled the
Hi Mahnoosh,
you might want to set bluefs_buffered_io to true for every OSD.
It looks it's false by default in v15.2.12
Thanks,
Igor
On 7/18/2021 11:19 PM, mahnoosh shahidi wrote:
We have a ceph cluster with 408 osds, 3 mons and 3 rgws. We updated our
cluster from nautilus 14.2.14 to octopu
