Hi, Zi Xuan After deep think, I have another question:
Why don't we combine ledger compaction and cursor compaction into one configuration switch? I’m not sure, do we have users who need to set the compression configuration for ledger and cursor separately? I think if they were to be set, they would all be set to the same. We could add new configurations ``xxxxxxxCompressionType`` to control the ledger and cursor compress and deprecate the ``managedLedgerInfoCompressionType`` for compatibility. After that, if users set ``managedLedgerInfoCompressionType``, we just compress the ledger, if the user set another configuration ``xxxxxxxCompressionType `` we will compress the ledger and cursor. It’s just a question or suggestion. You can feel free to go ahead. Best, Mattison > On Mar 6, 2022, at 8:22 AM, mattison chao <mattisonc...@gmail.com> wrote: > > > Great work! > > I have no other comments other than the compatibility everybody mentioned. > > Best, > Mattison > >> On Mar 6, 2022, at 4:55 AM, Enrico Olivelli <eolive...@gmail.com> wrote: >> >> Good proposal. >> It is important that this is disabled by default otherwise we cannot easily >> support the rollback >> >> Apart from that I don't have other comments >> >> >> >> Enrico >> >> Il Sab 5 Mar 2022, 11:22 PengHui Li <peng...@apache.org> ha scritto: >> >>> Hi Zixuan, >>> >>> We should add the compatibility part to the proposal. >>> And should also provide steps to roll back to the old version which >>> enabled the compression in the new version. >>> >>> I don't have objections to the proposal, and we have done >>> the same enhancement for topic metadata >>> https://github.com/apache/pulsar/pull/11490, >>> and this proposal also follows the same way. >>> >>> Thanks, >>> Penghui >>> >>> On Thu, Mar 3, 2022 at 10:26 AM Zixuan Liu <node...@gmail.com> wrote: >>> >>>> Thank you for your feedback. >>>> >>>> Forward compatibility is required: >>>> >>>> 1. Get CursorInfo from the metadata store: We will check the cursor data >>>> header, if it is compressed, we will parse the bytes data by compressed >>>> format, otherwise we will parse the cursor data directly by the original >>>> way. >>>> >>>> 2. Add/Update CursorInfo to the metadata store: The default is to use >>>> compression if the compression type is specified, otherwise we will put >>>> this data to the metadata store directly. >>>> >>>> >>>> >>>>> 2022年3月2日 下午11:48，Zike Yang <zky...@streamnative.io.INVALID> 写道： >>>>> >>>>> Hi, Zixuan >>>>> Thanks for creating this PIP. Here are my thoughts. >>>>> >>>>>> CursorInfo compression format >>>>>> >>>>>> [MAGIC_NUMBER] + [METADATA_SIZE] + [METADATA_PAYLOAD] + >>>> [MANAGED_CURSOR_INFO_PAYLOAD] >>>>>> >>>>>> MAGIC_NUMBER: Ox4779 >>>>> >>>>> Since we change the ManagedCursorInfo data format here. How do we >>>>> handle the old data format in the ZK. Could you explain the >>>>> compatibility for this PIP? >>>>> >>>>> Thanks, >>>>> Zike Yang >>>>> >>>>> On Wed, Mar 2, 2022 at 3:34 PM Zixuan Liu <node...@gmail.com <mailto: >>>> node...@gmail.com>> wrote: >>>>>> >>>>>> Hi Pulsar Community, >>>>>> >>>>>> I create a proposal that ManagedCursorInfo compression. The proposal >>>> can be found: https://github.com/apache/pulsar/issues/14529 < >>>> https://github.com/apache/pulsar/issues/14529> < >>>> https://github.com/apache/pulsar/issues/14529 < >>>> https://github.com/apache/pulsar/issues/14529>> >>>>>> >>>>>> Thanks, >>>>>> Zixuan >>>>>> >>>>>> ------------------ >>>>>> >>>>>> Motivation >>>>>> >>>>>> The cursor data is managed by ZooKeeper/etcd metadata store. When >>>> cursor data becomes more and more, the data size will increase and will >>>> take a lot of time to pull the data. Therefore, it is necessary to add >>>> compression for the cursor, which can reduce the size of data and reduce >>>> the time of pulling data. >>>>>> >>>>>> Goal >>>>>> >>>>>> Support use the LZ4/ZLIB/ZSTD/SNAPPY to compress the >>> ManagedCursorInfo. >>>>>> >>>>>> Implementation >>>>>> >>>>>> CursorInfo compression format >>>>>> >>>>>> [MAGIC_NUMBER] + [METADATA_SIZE] + [METADATA_PAYLOAD] + >>>> [MANAGED_CURSOR_INFO_PAYLOAD] >>>>>> >>>>>> MAGIC_NUMBER: Ox4779 >>>>>> >>>>>> METADATA >>>>>> Add a named ManagedCursorInfoMetadata message to MLDataFormats.proto >>>>>> >>>>>> message ManagedCursorInfoMetadata { >>>>>> required CompressionType compressionType = 1; >>>>>> required int32 uncompressedSize = 2; >>>>>> } >>>>>> CursorInfo compression and decompression design >>>>>> >>>>>> Currently, these compressions types have been defined and implemented >>>> by Pulsar, we only need to deal with compression and decompression of the >>>> ManagedCursorInfo data: >>>>>> >>>>>> Get CursorInfo from the metadata store >>>>>> We will check the cursor data header, if it is compressed, we will >>>> parse the bytes data by compressed format, otherwise by the original way. >>>>>> >>>>>> Add/Update CursorInfo to the metadata store >>>>>> The default is to use compression if the compression type is >>> specified. >>>>>> >>>>>> CursorInfo compression type configuration >>>>>> >>>>>> Add managedCursorInfoCompressionType in >>>> org.apache.pulsar.broker.ServiceConfiguration and >>>> org.apache.bookkeeper.mledger.ManagedLedgerFactoryConfig. >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Zike Yang >>>> >>>> >>> >
