Hi Mattison, Thanks for your feedback! I think using two configurations is more flexible, and users can set up different compression types.
Best, Zixuan mattison chao <mattisonc...@gmail.com> 于2022年3月6日周日 08:41写道： > Hi, Zi Xuan > > After deep think, I have another question: > > Why don't we combine ledger compaction and cursor compaction into one > configuration switch? > > I’m not sure, do we have users who need to set the compression > configuration for ledger and cursor separately? I think if they were to be > set, they would all be set to the same. > > > We could add new configurations ``xxxxxxxCompressionType`` to control the > ledger and cursor compress and deprecate the > ``managedLedgerInfoCompressionType`` for compatibility. > > After that, if users set ``managedLedgerInfoCompressionType``, we just > compress the ledger, if the user set another configuration > ``xxxxxxxCompressionType `` we will compress the ledger and cursor. > > > It’s just a question or suggestion. You can feel free to go ahead. > > Best, > Mattison > > > > > On Mar 6, 2022, at 8:22 AM, mattison chao <mattisonc...@gmail.com> > wrote: > > > > > > Great work! > > > > I have no other comments other than the compatibility everybody > mentioned. > > > > Best, > > Mattison > > > >> On Mar 6, 2022, at 4:55 AM, Enrico Olivelli <eolive...@gmail.com> > wrote: > >> > >> Good proposal. > >> It is important that this is disabled by default otherwise we cannot > easily > >> support the rollback > >> > >> Apart from that I don't have other comments > >> > >> > >> > >> Enrico > >> > >> Il Sab 5 Mar 2022, 11:22 PengHui Li <peng...@apache.org> ha scritto: > >> > >>> Hi Zixuan, > >>> > >>> We should add the compatibility part to the proposal. > >>> And should also provide steps to roll back to the old version which > >>> enabled the compression in the new version. > >>> > >>> I don't have objections to the proposal, and we have done > >>> the same enhancement for topic metadata > >>> https://github.com/apache/pulsar/pull/11490, > >>> and this proposal also follows the same way. > >>> > >>> Thanks, > >>> Penghui > >>> > >>> On Thu, Mar 3, 2022 at 10:26 AM Zixuan Liu <node...@gmail.com> wrote: > >>> > >>>> Thank you for your feedback. > >>>> > >>>> Forward compatibility is required: > >>>> > >>>> 1. Get CursorInfo from the metadata store: We will check the cursor > data > >>>> header, if it is compressed, we will parse the bytes data by > compressed > >>>> format, otherwise we will parse the cursor data directly by the > original > >>>> way. > >>>> > >>>> 2. Add/Update CursorInfo to the metadata store: The default is to use > >>>> compression if the compression type is specified, otherwise we will > put > >>>> this data to the metadata store directly. > >>>> > >>>> > >>>> > >>>>> 2022年3月2日 下午11:48，Zike Yang <zky...@streamnative.io.INVALID> 写道： > >>>>> > >>>>> Hi, Zixuan > >>>>> Thanks for creating this PIP. Here are my thoughts. > >>>>> > >>>>>> CursorInfo compression format > >>>>>> > >>>>>> [MAGIC_NUMBER] + [METADATA_SIZE] + [METADATA_PAYLOAD] + > >>>> [MANAGED_CURSOR_INFO_PAYLOAD] > >>>>>> > >>>>>> MAGIC_NUMBER: Ox4779 > >>>>> > >>>>> Since we change the ManagedCursorInfo data format here. How do we > >>>>> handle the old data format in the ZK. Could you explain the > >>>>> compatibility for this PIP? > >>>>> > >>>>> Thanks, > >>>>> Zike Yang > >>>>> > >>>>> On Wed, Mar 2, 2022 at 3:34 PM Zixuan Liu <node...@gmail.com > <mailto: > >>>> node...@gmail.com>> wrote: > >>>>>> > >>>>>> Hi Pulsar Community, > >>>>>> > >>>>>> I create a proposal that ManagedCursorInfo compression. The proposal > >>>> can be found: https://github.com/apache/pulsar/issues/14529 < > >>>> https://github.com/apache/pulsar/issues/14529> < > >>>> https://github.com/apache/pulsar/issues/14529 < > >>>> https://github.com/apache/pulsar/issues/14529>> > >>>>>> > >>>>>> Thanks, > >>>>>> Zixuan > >>>>>> > >>>>>> ------------------ > >>>>>> > >>>>>> Motivation > >>>>>> > >>>>>> The cursor data is managed by ZooKeeper/etcd metadata store. When > >>>> cursor data becomes more and more, the data size will increase and > will > >>>> take a lot of time to pull the data. Therefore, it is necessary to add > >>>> compression for the cursor, which can reduce the size of data and > reduce > >>>> the time of pulling data. > >>>>>> > >>>>>> Goal > >>>>>> > >>>>>> Support use the LZ4/ZLIB/ZSTD/SNAPPY to compress the > >>> ManagedCursorInfo. > >>>>>> > >>>>>> Implementation > >>>>>> > >>>>>> CursorInfo compression format > >>>>>> > >>>>>> [MAGIC_NUMBER] + [METADATA_SIZE] + [METADATA_PAYLOAD] + > >>>> [MANAGED_CURSOR_INFO_PAYLOAD] > >>>>>> > >>>>>> MAGIC_NUMBER: Ox4779 > >>>>>> > >>>>>> METADATA > >>>>>> Add a named ManagedCursorInfoMetadata message to MLDataFormats.proto > >>>>>> > >>>>>> message ManagedCursorInfoMetadata { > >>>>>> required CompressionType compressionType = 1; > >>>>>> required int32 uncompressedSize = 2; > >>>>>> } > >>>>>> CursorInfo compression and decompression design > >>>>>> > >>>>>> Currently, these compressions types have been defined and > implemented > >>>> by Pulsar, we only need to deal with compression and decompression of > the > >>>> ManagedCursorInfo data: > >>>>>> > >>>>>> Get CursorInfo from the metadata store > >>>>>> We will check the cursor data header, if it is compressed, we will > >>>> parse the bytes data by compressed format, otherwise by the original > way. > >>>>>> > >>>>>> Add/Update CursorInfo to the metadata store > >>>>>> The default is to use compression if the compression type is > >>> specified. > >>>>>> > >>>>>> CursorInfo compression type configuration > >>>>>> > >>>>>> Add managedCursorInfoCompressionType in > >>>> org.apache.pulsar.broker.ServiceConfiguration and > >>>> org.apache.bookkeeper.mledger.ManagedLedgerFactoryConfig. > >>>>>> > >>>>>> > >>>>> > >>>>> > >>>>> -- > >>>>> Zike Yang > >>>> > >>>> > >>> > > > >
