Hi Robert,
Thanks for pointing that out. This issue stems from version differences between
our cluster environments:
Our current Pacific cluster (migrating to Reef next week) uses the default
configuration:
bluestore_rocksdb_cfs = 'm(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru}
L P'
While Reef and newer versions use:
bluestore_rocksdb_cfs = 'm(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru}
L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32'
According to documentation, the bluestore_rocksdb_cfs parameter is only applied
during OSD creation (when using --mkfs). Since there was no mention of needing
to include L=min_write_buffer_number_to_merge=32
P=min_write_buffer_number_to_merge=32 in the reshard command, I didn't include
these options when resharding the OSDs. Did you?
Additionally, min_write_buffer_number_to_merge appears to be a RocksDB option
(bluestore_rocksdb_options) with a default value of 1 in Pacific and 6 in Reef.
It's unclear whether the effective value is determined by
bluestore_rocksdb_options or by the value specified in bluestore_rocksdb_cfs
during OSD creation.
This probably deserves clarification to prevent inconsistent behavior within
the same cluster due to OSDs having different min_write_buffer_number_to_merge
values from their creation time.
Hopefully Igor can provide some insight on this issue.
Regards,
Frédéric
----- Le 11 Mar 25, à 12:05, Robert Sander r.san...@heinlein-support.de a écrit
:
> Hi,
>
> Am 3/11/25 um 11:33 schrieb Frédéric Nass:
>
>> $ cephadm shell --name osd.OSD_ID --fsid $(ceph fsid) ceph-bluestore-tool
>> --path
>> /var/lib/ceph/osd/ceph-OSD_ID --sharding="m(3) p(3,0-12)
>> O(3,0-13)=block_cache={type=binned_lru} L P" reshard
>>
>> As a consequence, the 'Object' key/value pairs were stored in the default
>> column
>> family instead of the intended 'O' column family. Resharding again with the
>> correct capital 'O' resolved this issue for the affected overspilling OSDs.
>>
>> Be aware that there remains an error in both the Quincy and Reef
>> documentation,
>> where the letters 'l' and 'p' should actually be uppercase 'L' and 'P'. Using
>> lowercase 'l' and 'p' wrecks the OSD entirely.
>> I've created this tracker [1] in hopes of getting the Quincy and Reef
>> documentation fixed.
>
> The latest docs seems to have it correct:
>
> https://docs.ceph.com/en/latest/rados/configuration/bluestore-config-ref/#rocksdb-sharding
>
> Is there a reason why you omitted "=min_write_buffer_number_to_merge=32"
> after L
> and P?
> These are the default values that I also see on my reef cluster.
>
> Regards
> --
> Robert Sander
> Linux Consultant
>
> Heinlein Consulting GmbH
> Schwedter Str. 8/9b, 10119 Berlin
>
> https://www.heinlein-support.de
>
> Tel: +49 30 405051 - 0
> Fax: +49 30 405051 - 19
>
> Amtsgericht Berlin-Charlottenburg - HRB 220009 B
> Geschäftsführer: Peer Heinlein - Sitz: Berlin
> _______________________________________________
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
